Augmented reality based user interfacing

ABSTRACT

A display system renders a motion parallax view of object images based upon multiple observers. Also, a headset renders stereoscopic images that augment either physical objects viewed through the headset, or virtual objects projected by a stereoscopic display separate from the headset, or both. The headset includes a system for locating both physical objects and object images within a stereographic projection. Also, a proprioceptive user interface defines interface locations relative to body parts of an observer, the body parts include the head and the shoulders of the observer. The observer may enable a process associated with the interface location by indicating an intersection with the interface location with a physical object, such as a wand or a finger of the observer.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit under 35 USC § 119(e) to: U.S.patent application Ser. No. 14/905,698 filed Jan. 15, 2016 entitled“AUGMENTED REALITY BASED USER INTERFACING”, which claims priority to PCTApplication PCT/US14/47995 filed Jul. 24, 2014 and entitled “AUGMENTEDREALITY BASED USER INTERFACING”, which claims priority to U.S.Provisional Patent Application 61/859,264 filed Jul. 28, 2013 andentitled “MULTIPLE OBSERVER DISPLAY WITH MOTION PARALLAX”; U.S.Provisional Patent Application 61/878,257 filed Sep. 16, 2013 andentitled “AUGMENTATION OF A VIRTUAL REALITY OBJECT”; and U.S.Provisional Patent Application 61/911,020 filed Dec. 3, 2013 andentitled “PROPRIOCEPTIVE USER INTERFACE”, all of which are herebyincorporated by reference in their entirety.

FIELD

The present disclosure generally relates to stereoscopic userinterfaces, including augmentation of a virtual reality object projectedby a stereoscopic display.

BACKGROUND I. Motion Parallax

Motion parallax stereoscopic display systems are currently enabled inproducts such as the Learndo 3D terminal, the zSpace terminal, and EONReality's Ibench and Imobile devices. Motion parallax allows a virtualobject to appear stationary in real space when an observer changesperspective. Thus, even though an observer's head is moving, a virtualobject appears in a fixed space and the observer can “look around”surfaces of the virtual object. Motion parallax enhances the naturaluser interface experience afforded by the mixing virtual and realobjects in user interface and other applications. FIG. 1 shows anexample of a prior art motion parallax stereoscopic display system. Anobserver in a first position 100A is observing a stereoscopic display102 projecting a stereoscopic image of a first object image 104 locatedfront and center relative to the display 102 and a second object image106 located behind and on the left side relative to the display 102. Atracking system 106 has stereo cameras that track the position andmotion of the first observer. In order to make object images 104 and 106appear at their locations, a pair of stereoscopic images 114A and 116Aare rendered on the display 102 for the left and right eyes of theobserver in a manner known to those familiar with the art. A physicalobject 120A is operated by the first observer in order to interact withthe object images 104 and 106. The position and orientation of thephysical object is determined by tracking system 108 and in one example,a stereoscopic extension image 122A of the object is projected by thestereoscopic display. When an intersection of the stereoscopic extensionimage and the object image 104 or 106 is detected, an appropriateresponse is generated. One response may be to open an application suchas a word processing application. Such a system is described in U.S.Pat. No. 6,243,054 entitled Stereoscopic User Interface Method andApparatus, issued Jun. 5, 2001 to Michael DeLuca, and is herebyincorporated by reference.

When the first observer moves to a second position 100B, a motionparallax process is used to render the stereoscopic pair of images 114Band 116B to cause the first and second object images 104 and 106 toappear in substantially constant locations relative to the display 102even though the observer has moved to a different position. Physicalobject 120B and its stereoscopic extension image 122B are shown tocontinue to intersect object image 104 because the observer has changedthe position and orientation of the real object in response to theposition and motion of the observer. Display versions 102A and 102B showa two dimensional representation of the object images 104 and 106 viewedat positions 100A and 100B respectively. Object images 104A and 104Bappear larger than object images 106A and 106B because it is closer tothe observer. Display 102A shows object image 104A partially obscuringobject image 106A because of the observer's position 100A, while display102B shows object image 104B well to the right of object image 106Bbecause of the observer's position 100B. The system allows for thesimultaneous viewing of both virtual object images 104 and 106 and realphysical objects, including physical object 120 which is shown asextending beyond the display 102A and 102B as 120A and 120B. Display102A and 102B also show the stereoscopic extension image 122A and 122Bbeing rendered with motion parallax relative to the physical object.

Motion parallax provides for a natural user interface experience for thefirst observer. A second observer simply views the images as renderedfor the first observer, which results in an experience different fromthe natural user interface experience of the first observer.

II. Augmented Virtual Reality

Turning to the augmentation of a virtual reality object, head mounteddevices are being introduce that allow for augmentation of real objects.For example, U.S. Pat. No. 6,559,813 to DeLuca et al. shows a headsetthat displays augmented information to an observer also observing a realimage. The headset includes a camera system and a projector. The headsetcamera system monitors the physical objects also viewed by the observerwearing the headset, and the headset projector projects an image thataugments the physical objects. In one example, an observer is viewingthe Parthenon and a virtual image including additional informationregarding the Parthenon is projected by the headset so the observer isable to view both the Parthenon and augmented information regarding theParthenon. Such devices are currently being introduced includingGoogle's Google Glass, Atheer Labs' headset, and Meta's Space Glasses.

U.S. Pat. No. 6,559,813 also describes an image obstruction generatorfor blocking portions of a real image. In one example, the obstructiongenerator may block a bright portion of the real image, which mayenhance viewing of other real and/or virtual objects in the field ofview of the observer. Products which implement such an obstructiongenerator covered by the patent are being introduced and include thesunglasses made by Dynamic Eye.

U.S. Pat. No. 6,559,813 also describes a headset that projects astereoscopic display wherein physical objects, such as a finger or handof the observer, or a pointer held by the observer may interact withstereoscopic objects projected by the display based upon a determinedintersection between the physical object and the virtual object. Headsetproducts which implement intersections between virtual and real objectsare being introduced and include the Atheer Labs' headset and Meta'sSpace Glasses.

Thus, headsets have been provided that produce images that interact withreal objects within view of an observer. This is currently beingassociated with the term “natural user interface”. An observer may beable to view and interact with a stereoscopic image projected by aheadset projector as if the projected object image was a real object.However, headsets are incapable of interacting with a virtual objectimage projected by a stereoscopic display separate from the headset,such a display of a 3D movie, a zSpace terminal, or other stereoscopicprojection.

III. Proprioceptive User Interface

Turning now to proprioception, proprioception is a characteristicnatural to humans that describes an individual's sense of the relativepositions of neighboring parts of their body. For example, one's abilityto touch the nose on their face with their eyes closed or one's abilityto instinctively operate an automobile without looking at the footpedals, gear shifter, or steering wheel is attributed to proprioception.Much of proprioception is learned through visual feedback, and oncelearned, the individual can rely on muscle memory to complete anoperation, without requiring visual feedback. For example, after anindividual sees the location of automobile foot pedals and sees theirfeet operating the pedals, the location of the foot pedals is retainedin muscle memory and the individual may only need to occasionally lookat the pedal locations again to operate the automobile.

When an individual operates a graphical user interface to a computerizesystem, numerous interface icons may be rendered within the system'sdisplay area to allow a user to select which functionality of thecomputerized system is desired. A typical user interface home screen mayhave dozens or more interface icons for accessing a network, theInternet, often used applications, hard drive directories, file deletionfunctionality and other often used files. These icons have the benefitof allowing for quick access to processes associated with icons, howeverthey have a detriment in that they obscure the display area for otherrelevant information. For example, a computer automated design (CAD)application may be optimal if it occupies the entire area of a display,but if the user also needs occasional Internet access, either thedisplay area of the CAD application while using the application isreduced to allow for viewing of the Internet icon, thereby diminishingthe CAD application experience, or the user wanting to access theInternet while using the CAD application must be reduced or the displayarea of the CAD application window to reveal the Internet access icon,and then select the icon, thereby taking extra process steps andoperating time, ultimately diminishing the user interface experience.

A user of such a computer system moves frequently during its operation,and the location of the icons relative to the operator's body varieswidely, as such applications are not able to use proprioception toresolve the above stated problems. Furthermore, stereoscopic userinterfaces such as 3D terminals made Learndo 3D, zSpace, and EON Realityand the 3D terminal described in U.S. Pat. No. 6,559,813, describedetecting an intersection between a virtual object and a real object asperceived by an observer. Additionally, headsets from Atheer Labs andMeta and the 3D headset described in U.S. Pat. No. 6,559,813, alsodescribe detecting an intersection between a virtual object and a realobject in a space relative to an observer. Furthermore, portable displaydevices such as cellphones, tablets, eBooks, and video games are able tomove through a space relative to an observer while rendering augmentedimages relative to the space. Such devices ignore the natural humancharacteristic of proprioception in their user interfaces.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, and which together with the detailed description below areincorporated in and form part of the specification, serve to furtherillustrate various embodiments and to explain various principles andadvantages all in accordance with the present disclosure, in which:

FIG. 1 shows an example of a prior art motion parallax stereoscopicdisplay system;

FIG. 2A and FIG. 2B illustrate examples of a multiple observer displaywith motion parallax;

FIG. 3A and FIG. 3B illustrate an example of a motion parallax renderingof an object image shifting from a stationary first observer to a movingsecond observer;

FIG. 4A and FIG. 4B illustrate an example of a motion parallax renderingof an object image shifting from a moving first observer to a stationarysecond observer;

FIG. 5A and FIG. 5B illustrate an example of a motion parallax renderingof a first object image based upon a position of a first observer whilealso rendering a motion parallax rendering of a second object imagebased upon a position of a second observer;

FIG. 6 illustrates an example of a first physical object intersecting amotion parallax rendering of a first object image based upon a movingposition of a first observer while a second physical object intersects amotion parallax rendering of a second object image based upon a positionof a stationary second observer;

FIG. 7 illustrates an example of a motion parallax rendering of anobject image rendered as if it is attached to a moving physical objectof a first observer while being intersected by a physical object of asecond observer;

FIG. 8 illustrates an example of a block diagram of system implementinga multiple observer display with motion parallax;

FIG. 9 illustrates an example flow diagram of motion parallax displayrendered for a single observer;

FIG. 10 illustrates an example flow diagram of the rendering of a motionparallax display in response to both first and second observers;

FIG. 11 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesusing facial recognition for associating an object with an observer;

FIG. 12 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesand multiple stereoscopic extension images;

FIG. 13 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesand multiple stereoscopic extension images where motion parallaxrendering of multiple objects images transitions between observers;

FIG. 14 illustrates an example block diagram of a system foraugmentation of a virtual reality object;

FIG. 15 illustrates an example illustration of a headset worn by anobserver;

FIG. 16A and FIG. 16B illustrate an example of images viewed by anobserver of a stereoscopic projection display without and with theheadset of the disclosure;

FIG. 17A-FIG. 17D illustrate examples of augmentation of a virtualreality object image;

FIG. 18 illustrates an example of an intersection between an objectimage projected by a stereoscopic display and a stereoscopic imagerendered by a headset;

FIG. 19 illustrates a representative flow diagram for determining anintersection with an object image;

FIG. 20 illustrates a representative flow diagram for augmenting anobject image projected by a stereoscopic display with a stereoscopicimage rendered by a headset;

FIG. 21 illustrates a representative diagram of multiple observersobserving and intersecting an object image projected by a stereoscopicdisplay;

FIG. 22 illustrates and example of a flow diagram for determiningintersections with an object image projected by a stereoscopic displayand determined to be located in multiple locations relative to each ofthe multiple headsets;

FIG. 23 shows a side view of an observer having a proprioceptive userinterface;

FIG. 24A and FIG. 24B show a proprioceptive user interface havinginterface locations relative to various body parts that move with thevarious body parts;

FIG. 25A, FIG. 25B and FIG. 25C illustrate examples of limitedprojection space and imaging space of various devices able to implementa proprioceptive user interface;

FIG. 26 shows an example of a basic block diagram of the portable devicehaving a proprioceptive user interface;

FIG. 27A and FIG. 27B show an example of an operation of aproprioceptive user interface for an observer wearing a head mounteddevice such as a headset;

FIG. 28A and FIG. 28B show an example of an operation of aproprioceptive user interface with an observer viewing a stereoscopicterminal;

FIG. 29 shows a portable device having a display system and operating aproprioceptive user interface;

FIG. 30 shows object images rendered on the display of the portabledevice of FIG. 29 that augment the interface locations within thelimited display area of the portable device;

FIG. 31 shows an example of the portable device of FIG. 29 operating aproprioceptive user interface that is rendering an object imageaugmenting an interface location while acting as a physical object forindicating an intersection with the interface location located relativeto a body part of an observer;

FIG. 32 shows an example of the portable device of FIG. 29 operating aproprioceptive user interface that is not necessarily rendering anobject image augmenting an interface location while acting as a physicalobject for indicating an intersection with the interface locationlocated relative to a body part of an observer;

FIG. 33A and FIG. 33B show an example of a stereoscopic terminaloperating a proprioceptive user interface wherein an interface locationis moving with a body part of the observer and the observer isintersecting an interface location with either a left hand or a righthand;

FIG. 34 shows an example of an observer wearing a head mounted deviceobserving a reflection of the observer in a mirror with interfacelocations of the reflection augmented with object images rendered by thehead mounted device;

FIG. 35 shows an example of a perceived image of an observer wearing ahead mounted device looking at an observer reflection in a mirror,wherein the reflection is augmented with object images rendered by theheadset to indicate reflected interface locations;

FIG. 36 shows an example of an observer observing a display terminalrendering a mirror image of the observer with interface locations of themirror image augmented with object images rendered by the displayterminal;

FIG. 37 shows an example of a perceived image of an observer viewing adisplay terminal rendering a mirror image of the observer wherein themirror image of the observer is augmented with object images rendered bythe display terminal to indicate reflected interface locations;

FIG. 38 shows a block diagram illustration of a system for providing aportable proprioceptive user interface able to allow an observer tointerface with a number of devices using a common proprioceptive userinterface; and

FIG. 39 shows a representative flow diagram of a process for realizing aproprioceptive user interface.

DETAILED DESCRIPTION

Detailed embodiments are disclosed herein; however, it is to beunderstood that the disclosed embodiments are merely examples and thatthe systems and methods described below can be embodied in variousforms. Therefore, specific structural and functional details disclosedherein are not to be interpreted as limiting, but merely as a basis forthe claims and as a representative basis for teaching one skilled in theart to variously employ the disclosed subject matter in virtually anyappropriately detailed structure and function. Further, the terms andphrases used herein are not intended to be limiting, but rather, toprovide an understandable description.

I. Multiple Observer Display with Motion Parallax

FIG. 2A and FIG. 2B illustrate examples of a multiple observer displaywith motion parallax. In FIG. 2A, the parallax view of object images204A and 206A are rendered on display 202 with stereoscopic image pairs214A and 216A respectively in response to the position and motion of afirst observer 200A as determined by tracking system 208 as describedwith respect to FIG. 1. Because of the properties of stereoscopicrendering, the second observer 250A does not perceive the first objectin the same location relative to the display as the first observer 200A.Because both observers are observing the same stereoscopic image pairs214A and 216A rendered on display 202, the second observer 250Aperceives the first object ima1402ge at a location 254A while the firstobserver perceives the first object image at the location 204A.Similarly, the second observer 250A perceives the second object image ata location 256A while the first observer perceives the second object atthe location 206A. Furthermore, due to the motion parallax processes thefirst observer 200A perceives the first and second object images 204Aand 206A (shown in solid lines) to be substantially stationary relativeto the display 202 as the first observer motion results in a change inposition. However, the perceived locations of first and second objectimages 254A and 256A by the second observer (shown in dashed lines)change not only in response to the motion and position of the secondobserver because the motion parallax process is responsive to the firstobserver, but further in response to the motion and position of thefirst observer as the stereoscopic image pairs 214A and 216A arere-rendered by the motion parallax process to maintain a substantiallyconstant location relative to the display for the first observer 200A.

The tracking system 208 determines the position and orientation of thephysical object 220A operated by the first observer 200A and determinesan extension vector 222A. For example, as is known to those familiarwith the art, the real object may be a finger, a wand or pointer withfiducial marks to facilitate location by the tracking system, and inresponse an extension vector 222A is calculated to extend from thephysical object towards the display. A stereoscopic extension image maybe rendered based upon the extension vector. For example, thestereoscopic extension image may appear as a laser beam emanating fromthe physical object. In other examples, the physical object may be afinger of the observer and the stereoscopic image may be a paint brushtip or lightning bolt or may appear as an illuminated area whenintersecting a rendered object image. The first stereoscopic extensionimage is rendered for physical object 220A further in response to thefirst observer 200A motion parallax process. When an intersectionbetween the first observer's perceived location of an object image, suchas object image 204A intersecting extension vector 222A, then a processenabling response is enabled. The response could be launching of anapplication, such as a word processing application, or enable a motionor other modification of the object image 204A.

The tracking system 208 further determines the position and orientationof the second observer 250A and second physical object 270A operated bythe second observer 250A and determines a second extension vector 272A.The second stereoscopic extension image rendered for second physicalobject 270A is rendered with a second observer 250A motion parallaxprocess based upon the position and motion of the second observer. Whenan intersection between the second observer's perceived location of anobject image, such as object image 254A intersecting extension vector272A, then a process enabling response is enabled. The response could belaunching of an application, such as a word processing application, or amotion or other modification of the object image 204.

In another example, the extension vector is not required, and theintersection with the object image could correspond to the physicalobject itself intersecting with the object image. In this example, thephysical object could be a wand, pointer or an observer's finger. Whendetermining an intersection with the first object image, theintersection would be determined when the first observer's fingerintersected a first location corresponding to 204A, or the secondobserver's finger intersection a second location corresponding to 254A.After determining the intersection, the corresponding process can beenabled.

An example of a process being enabled in response to the second observer250A causing second physical object 270A to intersect with the perceivedlocation of the first object image 254A is to modify the motion parallaxprocess to render the first object image based upon the position andmotion of the second observer rather than the first observer. Thus, thesecond observer obtains motion parallax control of the first objectimage by intersecting it at its perceived location. The parallaxrendering of the second object image 256A can also be based upon theposition and motion of the second observer in response to the secondobserver causing the second physical object to intersect the firstobject image at its perceived location 254A. In another example, theparallax rendering of the second object image 256A can remain based uponthe position and motion of the first observer independent of the secondobserver causing the second physical object to intersect the firstobject image at its perceived location 254. In this example, the motionparallax view of the first object image is rendered based upon theposition and movement of the second observer and the second object imageis rendered based upon the position and motion of the first observer.

In these examples, the first stereoscopic extension image (if rendered)is rendered in response to both the position and motion of the firstobserver 200A and the position and orientation (and motion) of the firstreal object 220A. Further, the second stereoscopic extension image (ifrendered) is rendered in response to both the position and motion of thesecond observer 250A and the position and orientation (and motion) ofthe second real object 270A.

Display 202A shows the first observer's view of the display. Firstobject image 204A is in front of second object image 206A. Also notethat first object image 204A is rendered larger than second object image206A because first object image 204A is closer to the first observerthan the second object image, even though they may be similarly sized ina volumetric space defining the first and second object images. Thefirst physical object 220A and in this example a (laser beam) firststereoscopic extension image corresponding to first extension vector222A are also seen by the first observer. Furthermore, while the secondphysical object 270A may not be in the field of view of the firstobserver, in this example a (laser beam) second stereoscopic extensionimage corresponding to second extension vector 272A is seen by the firstobserver.

Display 202B shows the second observer's view of the display. Firstobject image 254A is in front of second object image 256A. The secondphysical object 270A and in this example a (laser beam) secondstereoscopic extension image corresponding to second extension vector272A are also seen by the second observer. Furthermore, while the firstphysical object 220A may not be in the field of view of the secondobserver, in this example the (laser beam) first stereoscopic extensionimage corresponding to first extension vector 222A is seen by the secondobserver.

In FIG. 2B, the parallax view of object images 254B and 256B arerendered on display 202 with stereoscopic image pairs 214B and 216Brespectively based upon the position and motion of the second observer250B as determined by tracking system 208. For purposes of illustration,first object image 254B corresponds to the same location first objectimage 204A relative to the display except that the motion parallaxperspective is rendered based upon the position and movement of thesecond observer 250B instead of the first observer 200A of FIG. 2A.Similarly, second object image 256B corresponds to the same locationfirst object image 206A except that the motion parallax perspective isrendered based upon the position and movement of the second observer250B rather than the first observer 200A of FIG. 2A. Correspondingly,first and second stereoscopic object image pairs 214B and 216B arerendered at different locations on the display 202 when compared tofirst and second object image pairs 214A and 216A of FIG. 2A.

Because of the properties of stereoscopic rendering, the first observer200B does not perceive the first and second objects in the same locationrelative to the display as the second observer 250B. Because bothobservers are observing the same stereoscopic image pairs 214B and 216Brendered on display 202, the first observer 200B perceives the firstobject image at a location 204B while the second observer perceives thefirst object image at the location 254B. Similarly, the second observer250B perceives the second object image at a location 256B while thefirst observer perceives the second object at the location 206B.Furthermore, due to the motion parallax processes the second observer250B perceives the first and second object images 254B and 256B (shownin solid lines) to be substantially stationary relative to the display202 as the second observer motion results in a change in position of thesecond observer. However, the perceived locations of first and secondobject images 204B and 206B by the first observer (shown in dashedlines) change not only in response to the motion and position of thefirst observer because the motion parallax process based upon the secondobserver, but further in response to the motion and position of thesecond observer as the stereoscopic image pairs 214B and 216B arere-rendered by the motion parallax process to maintain a substantiallyconstant location relative to the display for the second observer 250B.

The tracking system 208 determines the position and orientation of thephysical objects 220B and 270B operated by the first and secondobservers 200B and 250B respectively, and determines correspondingextension vectors 222B and 272B and enables rendering of correspondingstereoscopic extension images (if any) such a laser beams, arrows, endeffectors etc.

Display 202C shows the second observer's view of the display. Firstobject image 254B is to the right of second object image 256B. Also notethat first object image 254B is rendered larger than second object image265B because first object image 254B is closer to the second observerthan the second object image, even though they may be similarly sized ina volumetric space defining the first and second object images. Thesecond physical object 270B and in this example a (laser beam) secondstereoscopic extension image corresponding to second extension vector272B are also seen by the second observer. Furthermore, while the firstphysical object 220B may not be in the field of view of the secondobserver, in this example a (laser beam) first stereoscopic extensionimage corresponding to second extension vector 222B is seen by thesecond observer.

Display 202D shows the first observer's view of the display. Firstobject image 204B is to the right of second object image 206B. Thesecond physical object 220B and in this example a (laser beam) secondstereoscopic extension image corresponding to second extension vector222B are also seen by the first observer. Furthermore, while the secondphysical object 270B may not be in the field of view of the firstobserver, in this example the (laser beam) second stereoscopic extensionimage corresponding to first extension vector 272B is seen by the firstobserver.

In FIG. 2A, the first and second object images are rendered with amotion parallax process based upon the first observer and in FIG. 2B thefirst and second object images are rendered with the motion parallaxprocess based upon the second observer. The tracking system tracks boththe first and second observers and their physical objects and determinesan intersection where the object image is perceived, even though thefirst and second observers perceive the object images in differentlocations relative to the display and an object image is rendered usinga motion parallax process based upon the position and location of one ofthe observers. In response to the determined intersection, any of anumber of processes may be enabled, including modifying the motionparallax rendering of the intersected object image such that it isrendered be based upon the observer causing the intersection.

The tracking system can determine that first physical object 220A isbeing operated or controlled by the first observer 200 by opticallyanalyzing tracking system imaging to determine the physical object isheld or in contact with the first observer. Alternately tracking system208 can distinguish physical object 220A and 220B and associate physicalfirst object 220A with the first observer 200A in response to apredetermined association or a facial recognition of the first observer.Other known approaches for associating a physical object with anobserver are considered to be included in the scope of the disclosure.

FIG. 3A and FIG. 3B illustrate an example of a motion parallax renderingof an object image shifting from a stationary first observer to a movingsecond observer. Rendered on display 302 is a stereoscopic image pair314 that results in projection of an object image at location 354 fromthe perspective of a stationary first observer 300. The motion parallaxrendering of the object image is based upon the first observer, asindicated by the object image 354 of FIG. 3A having no internal hashingmarks. Consequently, a second moving observer moving from positions 350Ato position 350B perceives the object image location as varying fromlocation 354A to 354B. If while at position 350B, the second observerintersects the object image at location 354B with a real object 370 andan extension vector 372, then the motion parallax rendering of theobject image shifts from being based upon the position and movement ofthe first observer to the position and movement of the second observer.FIG. 3B shows the second observer moving from position 350B to position350C, which in this example corresponds to moving back to position 350Aof FIG. 3A. The perceived location 354B of the object image is not shownto change for the second observer in FIG. 3B because motion parallaxrendering of the object image is based upon the position and motion ofthe second observer, as indicated by the object image 354B of FIG. 3Bhaving no internal hashing marks. However, the first observer doesperceive the object image as changing from location 354 to location 354Cin response to the motion and position of the second observer.

FIG. 3A and FIG. 3B show an example of enabling a physical object 370 tobe observable by a second observer 350A, 350B, 350C in the space inaddition to an object image 354A. 354B, determining the physical objectindicating an intersection 372, 354B with the variable locationprojection 354A, 354B of the object image, and generating an inputsignal based upon the determining that enables the motion parallaxrendering the process to render the object image based upon the secondobserver instead of the first observer.

FIG. 4A and FIG. 4B illustrate an example of a motion parallax renderingof an object image shifting from a moving first observer to a stationarysecond observer. Rendered on display 402 is a stereoscopic image pair414A that result in projection of an object image at location 454 fromthe perspective of a first observer moving from 400A to 400B. The motionparallax rendering of the object image is based upon the first observer,as indicated by the object image 454 of FIG. 4A having no internalhashing marks. In response to the first observer moving from position400A to position 400B, the rendering of the stereoscopic image paircorrespondingly moves from 414A to 414B to provide for the motionparallax view of the object image based upon the first observer.Consequently, a second stationary observer at position 450 perceives theobject image location as varying from location 454A to 454B in responseto the first observer moving between positions 400A to 400B. If whilethe first observer is at position 400B, the second observer intersectsthe object image at location 454B with a real object 470 and anextension vector 472, then the motion parallax rendering of the objectimage shifts from being based upon the position and movement of thefirst observer to the position and movement of the second observer. FIG.4B shows the first observer moving from position 400B to position 400C,which in this example corresponds to moving back to position 400A ofFIG. 4A. The perceived location 454B of the object image is not shown tochange for the second observer in FIG. 4B because motion parallaxrendering of the object image is based upon the position and motion ofthe second observer, as indicated by the object image 454B of FIG. 4Bhaving no internal hashing marks. However, the first observer doesperceive the object image as changing from location 454 to location 454Cin response to the motion and position of the first observer.

The input signal that caused the motion parallax rendering of the objectto shift from the first observer to the second observer in FIG. 4A andFIG. 4B is shown as the extension vector 472 intersecting the objectimage at location 454B. In other examples, other inputs can cause theshift. For example, physical object itself 470, or the finger or otherbody part of the second observer could intersect the object image atlocation 454B. Alternately a command could be entered using a keyboardor mouse or other input device associated with the display. In anotherexample, the shift can be based upon a facial recognition process.Facial recognition processes are known to those familiar with the art.The illustration shows the first observer having circular facialcharacteristics and the second observer having angular facialcharacteristics. An optical system can analyze the various facialcharacteristics to distinguish observers and either identify an observeror associate and observer with a database of known observers. If theappropriate privileges are associated with the second observer foraccessing and or controlling processes associated with a system coupledto the display, then the motion parallax rendering of an object imagecan shift to the second observer based upon facial recognition. With anoptical tracking system, both the position and movement and facialrecognition of the observers can be processed. This can enablegeneration of the input signal for switching motion parallax renderingof an object from one observer to another observer. Furthermore, theoptics of the tracking system can be used to process the images todetermine positions, orientations and movements of observers andphysical objects associated with the observers in addition to facialrecognition. For example if one observer is a manger of a secondobserver, then motion parallax rendering may be shifted to the managerusing such an approach, however a subordinate may not have theprivileges to shift motion parallax rendering away from the manager withthis approach.

FIG. 5A and FIG. 5B illustrate an example of a motion parallax renderingof a first object image based upon a position of a first observer whilealso rendering a motion parallax rendering of a second object imagebased upon a position of a second observer. In FIG. 5A, the first objectimage is projected at location 506 as the first observer moves fromposition 500A to position 500B which results in first object image pairsto be rendered at 516A and 516B on display 502 in response. The secondobserver is at stationary position 550 and observes the first objectimage to be projected at locations 506A and 506B in response to themovement of the first observer. The second object image is secondobserver motion parallax projected at location 554 with stereoscopicimage pair 514. The second observer is at stationary position 550 andobserves the second object image to be projected at locations 554independent of the movement of the first observer. As the first observermoves from position 500A to position 500B, the first observer views thesecond object image as appearing to move from location 554A to 554B inresponse, while the first object image appears stationary at location506. The second observer views the second object image as appearingstationary at location 554, while the first object image appears to movefrom location 506A to 506B in response to the first observer movementand position.

In FIG. 5B, the second object image is projected at location 554C as thesecond observer moves from position 550C to position 550D which resultsin second object image pairs to be rendered at 514C and 514D on display502. The first observer is at stationary position 500C and observes thesecond object image to be projected at locations 554C and 554D inresponse to the movement of the second observer. The first object imageis first observer motion parallax projected at location 506C withstereoscopic image pair 516C. The first observer is at stationaryposition 500C and observes the first object image to be projected atlocation 506C independent of the movement of the second observer. As thesecond observer moves from position 550C to position 550D, the secondobserver views the first object image as appearing to move from location506C to 506D in response, while the second object image appearsstationary at location 554C. The first observer views the first objectimage as appearing stationary at location 506C, while the second objectimage appears to move from location 554C to 554D in response to thesecond observer movement between position 500C and 500D.

FIG. 5A and FIG. 5B show a display where some object images are motionparallax rendered for one observer and other object images are motionparallax rendered for another observer. Object images motion parallaxrendered for an observer appear at a constant location relative to thedisplay for the observer even though any observer may move positions.Object images not motion parallax rendered for the observer appear atchange location relative to the display dependent upon motion andposition of other observers. If an object image is associated with anobserver, then motion parallax rendering the object based upon theobserver is a natural indication that the object is associated with theobserver because it appears at a relatively stationary location relativeto the display.

FIG. 6 illustrates an example of a first physical object intersecting amotion parallax rendering of a first object image based upon a movingposition of a first observer while a second physical object intersects amotion parallax rendering of a second object image based upon a positionof a stationary second observer. In FIG. 6, the first object image isprojected at location 606 as the first observer moves from position 600Ato position 600B which results in first object image pairs to berendered at 616A and 616B on display 602 in response. The secondobserver is at stationary position 650 and observes the first objectimage to be projected at locations 606A and 606B in response to themovement of the first observer. The second object image is secondobserver motion parallax projected at location 654 with stereoscopicimage pair 614. The second observer is at stationary position 650 andobserves the second object image to be projected at locations 654independent of the movement of the first observer. As the first observermoves from position 600A to position 600B, the first observer views thesecond object image as appearing to move from location 654A to 654B inresponse, while the first object image appears stationary at location606, and the second observer views the second object image as appearingstationary at location 654, while the first object image appears to movefrom location 606A to 606B in response to the first observer movementand position. The first physical object of the first observer is shownas a finger of the first observer. In first observer position 600A,finger 602A intersects the first object image located at 606 and thenthe first observer moves to position 600B, finger 602B intersects thefirst object image at the same location 606, even though the firstobserver has moved to a different position. The second physical objectof the second observer 650 is shown as the finger 652 of the secondobserver, which is shown to intersect the second object image atlocation 654 independent of the position of the first observer. Thus,each observer observes a motion parallax rendered object in a constantlocation relative to the display independent of the motion of eitherobserver. The intersection of the constant location with a physicalobject results in generation of a signal which enables subsequentprocesses.

FIG. 7 illustrates an example of a motion parallax rendering of anobject image rendered as if it is attached to a moving physical objectof a first observer while being intersected by a physical object of asecond observer. First observer 700A controls a first physical object702A which is shown as a finger of the first observer. An object image706A is rendered with motion parallax with stereoscopic image pairs 716Aon display 702 to appear as if the object image is attached to thefinger of the first observer. The observer moves to position 700B andmoves physical object finger to position 706B. The tracking system (notshown) determines the position of the observer 702B and the position ofthe first observer's finger 702B and renders stereoscopic image pair716B on display 702 to project the first object image in a location 706Bto correspond to the position of the finger based upon the position ofthe observer. Thus, the first object image appears to the first observeras if it is attached to the finger of the first observer. The secondobserver is in a stationary position 750 and perceives the first objectat location 706C in response to the first observer having a position700A and a finger at position at 702A. The second observer alsoperceives the first object at location 706D in response to the firstobserver having a position 700B and a finger at position 702B, and thephysical object controlled by the second observer, shown as a finger ofthe second observer 752 is placed at location 706D to cause anintersection with the perceived location of the object image. An inputsignal is generated in response and a subsequent process may be enabled.One example of a subsequent process would be to render the object imageas if it were attached to the finger of the second observer, therebytransferring motion parallax rendering of the object image based uponthe second observer and second physical object, rather than the firstobserver and the first physical object.

FIG. 8 shows an example of a block diagram of system implementing amultiple observer display with motion parallax. Tracking system 820receives information regarding a first observer 800 and a physicalobject 802 controlled by the first observer as well as a second observer810 and a physical object 812 controlled by the second observer.Tracking system is able to produce image data including first observerimage data and second observer image data allowing unique identificationof the first and/or the second observers using facial recognition ofother approach as well as the ability to determine the position of thefirst and second observers, and their motion by determining time-to-timechanges in position. Identification of the observers also allows forassociation of object images with observers. Tracking system 820 may beany number of tracking systems known to those familiar with the art andmay include one or more pairs of stereo cameras receiving light in theinfrared, visible and/or ultraviolet spectrum in order to obtain imagestracking multiple observers and multiple objects. In one example, thetracking may be simplified by placing fiducial dots approximate to theeyes of the observers and approximate to the physical object. The eachfiducial dot may be highly reflective to light or a certain spectrum andthe tracking system may include corresponding light radiators toilluminate the fiducial dots for the cameras. Such systems for detectingsingle observers and objects have been described in aforementioned U.S.Pat. No. 6,243,054 and implemented by zSpace and Leonardo 3D devices. Inanother example, the tracking system can radiate one or more arraypattern of light and receive the pattern with one or more cameras inorder to process the pattern to detect observers and physical objects.Such a system has been implemented by the Microsoft Kinect. Such systemsmay include a time of flight dimension to facilitate determining adistance of an observer or an object from the tracking system. Facialrecognition processor 822 process images received by the tracking systemto identify observers. Facial recognition processes are known to thosefamiliar with the art. Once identified, privileges associated with theobservers may be utilized.

The first observer detector 824 and the second observer detector 826process the data from the tracking system to determine the position,orientation and motion of the first and second observers. The firstphysical object detector 828 and the second physical object detector 830process the data from the tracking system to determine the position,orientation and motion of the first and second physical objects.Volumetric data 832 includes three dimensional data about objects to berendered in a space about the display. Individual object images aredetermined from the volumetric data by first object imaging module 834and second object imaging module 836. Upon displaying the data theobjects are rendered with object images relative to the display 840, thefirst observer 800, the first physical object 802, the second observer810 and the second physical object 812. The first extension image module842 determines a first extension vector based upon the position,orientation and motion of the first physical object 802 (and acorresponding first extension image if appropriate) and the secondextension image module 844 determines a second extension vector basedupon the position, orientation and motion of the second physical object812 (and a corresponding second extension image if appropriate).

The first motion parallax module 846 determines how to render a motionparallax view of object images associated with the first observer 800based upon the position, motion and orientation of the first observer.Object images associated with the first observer can include the firstextension image, the first object image and/or the second object imageas appropriate. The second motion parallax module 848 determines how torender a motion parallax view of object images associated with thesecond observer 810 based upon the position, motion and orientation ofthe second observer. Object images associated with the second observercan include the second extension image, the first object image and/orthe second object image as appropriate. Controller 850 controls thesystem and, among other things, determines which objects andcorresponding object images and extension images are associated with thefirst and second observers. Controller 850 may include a computer and anon-transitory computer readable medium having a stored set ofinstructions that when executed cause the system to operate as describedherein.

The parallax rendering module 852 receives information regarding thefirst object image from first object imaging module 834, the secondobject image from the second object image module 836, the firstextension image from the first extension image module 842, the secondextension image from the second extension image module 844, the firstobserver motion parallax for first observer motion parallax module 846and the second observer motion parallax perspective from the secondobserver motion parallax module 848 to render images for the display840. Stereoscopic rendering module 584 determines how to render left andright eye image pairs of the object images for a stereoscopic projectionof the object images. It should be noted that in some examples of motionparallax, stereoscopic projection of the rendered image is not necessaryand only a single left eye or right eye version of the image is renderedwhile displaying a motion parallax rendering. The stereoscopic (ornon-stereoscopic single eye) image is rendered on display 840. Display840 may project stereoscopic images in any manner known to thosefamiliar with the art included polarizing or active shutter glassessystem and glasses free systems including parallax projecting displays.Also, display 840 can be a non-stereoscopic display if single eyenon-stereoscopic motion parallax renderings are to be displayed.

First intersection detector module 860 detects if an intersection hasbeen indicated by an observer controlling a physical object and viewingthe first object image. If the first object image is rendered withmotion parallax based upon the first observer, then the location of thefirst object image perceived by the first observer will appear in asubstantially constant location relative to the display, while thelocation of the first object image perceived by the second observer willbe in a variable location depending on the position and movement of thefirst and second observers. If the first object image is rendered withmotion parallax based upon the second observer, then the location of thefirst object image perceived by the second observer will be in asubstantially constant location relative to the display, while thelocation of the first object image perceived by the first observer willbe in a variable depending on the position and movement of the first andsecond observers. A first input signal is generated in response to thedetection of the intersection. The first input signal can include anindication that the intersection was caused by a physical objectcontrolled or operated by either the first or the second observer.

Second intersection detector module 862 detects if an intersection hasbeen indicated by an observer controlling a physical object and viewingthe second object image. If the second object image is rendered withmotion parallax based upon the first observer, then the location of thesecond object image perceived by the first observer will appear in asubstantially constant location relative to the display, while thelocation of the second object image perceived by the second observerwill be in a variable location depending on the position and movement ofthe first and second observers. If the second object image is renderedwith motion parallax based upon the second observer, then the locationof the second object image perceived by the second observer will appearin a substantially constant location relative to the display, while thelocation of the second object image perceived by the first observer willbe in a variable location depending on the position and movement of thefirst and second observers. A second input signal is generated inresponse to the detection of the intersection. The second input signalcan include an indication that the intersection was caused by a physicalobject controlled or operated by either the first or the secondobserver.

Controller 850 receives the input signals and enables processes inresponse. In one example, an enabled process sends a signal toinput/output 870 which may cause generation of an audio alert or maycommunicate the signal to another device to enable a further processsuch as turning off or on or otherwise controlling a piece of equipment.In another example, an enabled process can include launching anapplication 880 in the device, such as a word processing or emailapplication or any of numerous other applications known to thosefamiliar with the art. In another example the enabled process modifiesthe object in the volumetric data base 832, modifications includechanging an objects size, position, velocity and direction (if theobject is moving within the volumetric database) or other characteristicof the object. In another example, the enabled process can change themotion parallax perspective of the object image by changing the observerupon which the motion parallax of the object image is rendered.

FIG. 9 illustrates an example flow diagram of motion parallax displayrendered for a single observer only. Step 902 determines the positionand motion of a first observer and step 904 renders a motion parallaxview of an object image based upon the first observer. If a secondobserver is detected at step 906, then step 908 renders the view of theobject image without motion parallax. Thus, if more than one observer isdetected, then the flow diagram terminates rendering of a motionparallax view of the object image. Note that in the flow diagram of FIG.9 the object image may or may not be stereoscopically projected. In someapplications, rendering of a motion parallax object image may beundesirable by the observer for which motion parallax is not based upon.Thus, rendering the object image without motion parallax may be abenefit when multiple observers are detected.

FIG. 10 illustrates an example flow diagram of the rendering of a motionparallax display in response to both first and second observers. Step1000 determines the position of a first observer. Step 1002 renders amotion parallax view of an object image for the first observer. Step1004 determines if a second observer is detected. If so, step 1006determines the position and motion of the second observer and step 1008combines the positions and motions of the first and second observers.Step 1010 renders a motion parallax view of the object image based uponthe combined positions and motions of the first and second observers.The combination of the positions and motions of the first and secondobservers may result in the rendering of motion parallax from a thirdsynthesized observer which in one example exhibits a position and motionof the average position and motion of the first and second observers. Inanother example, the position and motion of one observer may be givenmore weight than the other observer. For example, if one observer hadrecently caused a physical object to intersect a location of an objectimage, then that observer's position and movement may be given twice theweight of the other observer. If for example X, Y and Z representCartesian coordinates relative to a space in front of the display, andthe position of observer one relative to the display is X1, Y1, Z1, andthe position of the other observer is X2, Y2, Z2 and the position of thesynthesized observer is X3, Y3 and Z3, then in the first example,X3=(X1+X2)/2, Y=(Y1+Y2)/2, Z3=(Z1+Z2)/2, while in the second examplewhere one observer has a greater weight, then X3=(2*X1+X2)/3,Y3=(2*Y1+Y2)/3, Z3=(2*Z1+Z2)/3. In another example, a preferredobservation position of X0, Y0, Z0, can be included in the determinationwhere X3=(X0+X1+X2)/3, Y3=(Y0+Y1+Y2)/3, Z3=(Z0+Z1+Z2)/3 where thepreferred observer position can correspond to a location with optimumstereoscopic viewing such as a position centered about the display witha preferred distance from the display, say eighteen inches for example.

FIG. 11 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesusing facial recognition for associating an object image with anobserver. Step 1100 determines the position and motion of the firstobserver. Step 1102 renders a motion parallax view of a first objectimage and a second object image based upon the first observer. Step 1104determines if a second observer has been detected with facialrecognition, or identified with another input signal, as previouslydiscussed. If so, step 1106 associates the first object image with thefirst observer and the second object image with the second observer. Theassociation of the second object image with the second observer may bemade based upon a predetermined linkage between second object image andthe second observer in response to identification of the observer byeither facial recognition or other input signal. Step 1108 determinesthe position and motion of both the first and second observers and step1110 renders a motion parallax view of the first object image based uponthe first observer and step 1112 renders a motion parallax view of thesecond object image based upon the second observer. FIG. 11 shows anexample of associating multiple object images with multiple observersand rendering the object images with motion parallax based upon eachassociated observer. Building on this example, one set of object images,such as a menu of items can be associated with a first observer, whereineach item in the menu corresponds to a process enabling object imageassociated with the first observer while a second menu of items can beassociated with the second observer. Each item menu associated with thecorresponding observer will appear relatively stationary on the displayfor the associated observer, independent of the position and movement ofeither observer. This provides an enhanced user interface for bothobservers viewing the display.

FIG. 12 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesand multiple stereoscopic extension images. Step 1202 determines theposition and motion of first and second observers. Step 1204 renders amotion parallax view of the first object image based upon the firstobserver. Step 1206 renders a motion parallax view of the second objectimage based upon the second observer. Step 1208 determines the positionand orientation of first and second physical objects operated by thefirst and second observers. Step 1210 determines the first and secondextension vectors associated with the first and second physical objectsand their positions, orientations and motions. Step 1212 renders amotion parallax view of the first stereoscopic extension image relativeto the first object based upon the first observer. Step 1214 renders amotion parallax view of the second stereoscopic extension image relativeto the second object based upon the second observer. Step 1216determines the projected location of the first and second object imagesas viewed by the first observer. Step 1218 determines if either thefirst object, the first extension vector, or the first extension imageintersects with either the first or second projected locations of thefirst or second object images based upon the first observer. If anintersection is detected, then an input signal is generated and any of anumber of processes can be enabled. A first or second process can beenabled in response to an intersection with the first or second objectimage locations. Also, the first or second object images can bemodified, a first or second application can be launched, a first orsecond audio signal can be generated or a first or second output signalcan be generated. Also, if the intersection is with the location of thesecond object image, then the motion parallax view of the second objectimage can be rendered based upon the first observer in response to thefirst observer. It should be appreciated that although steps 1216-1220are shown with respect to the first observer, a similar set of steps canbe taken with respect to the second observer.

In modification, the parallax rendering of a physical object which isbased upon observer can be a component in determining if an intersectioncaused by physical element or object is qualified to result ininitiating of a subsequent process. In one example, only intersectionscaused by an observer upon which motion parallax is rendered for theobject image will be a qualified intersection. Thus, only the observerfor whom the object image appears stable can cause a qualifiedintersection with the object image. In examples with a multiplicity ofobservers, only a plurality of observers may cause a qualifiedintersection. The plurality of observers may be identified in responseto facial recognition or the selective process.

FIG. 13 illustrates and example flow diagram of the rendering of amultiple observer motion parallax display with multiple object imagesand multiple stereoscopic extension images where motion parallaxrendering of multiple objects images transitions between observers. Step1302 determines the position and motion of the first and secondobservers. Step 1304 renders a motion parallax view of a first objectimage, an associated object image associated with the first objectimage, and another object image based upon the first observer. Step 1306renders a stereoscopic motion parallax view of a second object imagebased upon the second observer. Step 1308 determines the position andorientation of first and second physical objects operated by first andsecond observers. Step 1310 determines first and second physical objectextension vectors based upon the determined position and orientation ofthe physical objects. Step 1312 renders a stereoscopic motion parallaxview of the first extension image of the first physical object basedupon the first observer. Step 1314 renders a stereoscopic motionparallax view of the second extension image of the second physicalobject based upon the second observer. Step 1316 determines the firstprojected location of the first object image as viewed by the secondobserver. Step 1318 determines if the second physical object or thesecond extension vector or the second extension image intersects withthe first projected location.

If so, step 1320 determines the position and motion of the first andsecond observers. Step 1322 renders a motion parallax view of the otherobject image based upon the first observer. Step 1324 renders astereoscopic motion parallax view of the first object image, theassociated object image associated with the first object image, and thesecond object image based upon the second observer. Thus the motionparallax view of the first object image and the associated object imageare rendered based upon the second observer in response to anaffirmative result of step 1318. Step 1326 determines the position andorientation of first and second physical object operated by first andsecond observers. Step 1328 determines first and second physical objectextension vectors based upon the determined position and orientation ofthe physical objects. Step 1330 renders a stereoscopic motion parallaxview of the first extension image of the first physical object basedupon the first observer. Step 1332 renders a stereoscopic motionparallax view of the second extension image of the second physicalobject based upon the second observer. Step 1334 determines theprojected location of the first object image as viewed by the firstobserver. Step 1336 determines if the first physical object or the firstextension vector or the first extension image intersects with the firstprojected location. If so, then the flow diagram returns to step 1302 torender the first object image and associated image with a motionparallax view based upon the first observer, otherwise the flow diagramreturns to step 1320 to render the first object image and associatedimage with a motion parallax view based upon the second observer.

One of the operations shown by FIG. 13 is that the motion parallaxrendering of an object image and an associated object image can betransferred between observers. The transfer occurs in response to anobserver causing an intersection with the object image. The object imageand the associated object image may be included in a menu of items,where an intersection with each object image in the menu enables acorresponding process.

While the object images are shown in the figures as stationary, thosefamiliar with the art will appreciate that the object images need not bestationary and may have a motion associated with them while remainingwithin the scope of the disclosure.

While the disclosure describes multiple viewer motion parallax systemwith respect to two observers, those familiar with the art willappreciate that the system may be extended beyond two observers, whereeach observer can have an object image rendered with motion parallaxbased upon the observer. Additionally, the system can determine aphysical object based intersection with an object image based upon anobserver position, when the object image is not rendered with motionparallax based upon the observer controlling the physical object.

The present subject matter can be realized in hardware, software, or acombination of hardware and software. A system can be realized in acentralized fashion in one computer system, or in a distributed fashionwhere different elements are spread across several interconnectedcomputer systems. Any kind of computer system, or other apparatusadapted for carrying out the methods described herein, is suitable. Atypical combination of hardware and software could be a general purposecomputer system with a computer program that, when being loaded andexecuted, controls the computer system such that it carries out themethods described herein.

In a summary, the disclosure includes a method comprising: enabling aprocess to render on a display a first observer motion parallax view ofan object image based upon a first observer, the rendered object imagesimultaneously viewable by the first observer and a second observer; andenabling the process to render on the display a second observer motionparallax view of the object image based upon the second observer and aninput signal. The process renders the first observer motion parallaxview based upon a position and a motion of the first observer and theprocess renders the second observer motion parallax view based upon aposition and a motion of the second observer. The method furthercomprising: receiving a second observer image of the second observer;recognizing a distinguishing element in the second observer imageindicative of the second observer; and generating the input signal basedupon the recognizing. The method further comprising determining aposition of the second observer based upon the second observer imagewherein the process renders the second observer motion parallax viewbased upon the position of the second observer. The method furthercomprising: determining a position and a motion of the second observerwhile the process is enabled to render the first observer motionparallax view of the object image; and determining a position and amotion of the first observer while the process is enabled to render thesecond observer motion parallax view of the object image. The objectimage is a stereoscopic projection in a space relative to the displayand rendering of the first observer motion parallax view results in asubstantially constant location projection of the object image in thespace relative to the first observer based upon the motion of the firstobserver and a variable location projection of the object image in thespace relative to the second observer based upon the motion of the firstobserver, and the method further comprises: enabling a physical objectto be observable by the second observer in the space in addition to theobject image; determining the physical object indicating an intersectionwith the variable location projection of the object image; andgenerating the input signal based upon the determining of theintersection. The generating further comprises generating the inputsignal based upon a determination of the physical object being operatedby the second observer. The method further comprising: determining aposition and an orientation of the physical object; and determining anextension vector of the physical object based upon the determinedposition and orientation of the physical object, wherein the determiningthe physical object indicating the intersection with variable locationprojection of the object image further includes determining anintersection of the extension vector with the variable locationprojection of the object image. The enabling the process to render thefirst observer motion parallax view of the object image based upon thefirst observer includes enabling the process to render on the display asecond observer motion parallax view of a stereoscopic extension imagebased upon the second observer and the extension vector. The enablingthe process to render the first observer motion parallax view of theobject image based upon the first observer also enables the process torender on the display a first observer motion parallax view of a secondobject image based upon the first observer; and the enabling the processto render on the display the second observer motion parallax view of theobject image based upon the second observer and the input signal alsoenables the process to render on the display the first observer motionparallax view of the second object image based upon the first observer.The method further comprising: the enabling the process to render on thedisplay the first observer motion parallax view of the object imagebased upon the first observer, also enables the process to render on thedisplay a first observer motion parallax view of a second object imagebased upon the first observer; and the enabling the process to render onthe display the second observer motion parallax view of the object imagebased upon the second observer and the input signal also enables theprocess to render on the display a second observer motion parallax viewof the second object image based upon the second observer.

In a summary, the disclosure also includes a method comprising:rendering a first observer motion parallax view of a first object imageon a display based upon a first observer; and rendering a secondobserver motion parallax view of a second object image on the displaybased upon a second observer also able to view rendering of the firstobserver motion parallax rendering of the first object image on thedisplay based upon the first observer. The method further comprising:receiving image data including first observer image data and secondobserver image data; determining a position of the first observer basedupon the first observer image data; uniquely identifying the firstobserver based upon the first observer image data; and associating thefirst object image with the first observer based upon the uniquelyidentifying the first observer; wherein the first observer motionparallax view is rendered based upon the associating the first objectimage with the first observer and the determining the position of thefirst observer. The method further comprising: determining a position ofthe second observer based upon the second observer image data; uniquelyidentifying the second observer based upon the second observer imagedata; and associating the second object image with the second observerbased upon the uniquely identifying the second observer, wherein thesecond observer motion parallax view is rendered based upon theassociating the second object image with the second observer and thedetermining the position of the second observer. The method furthercomprising: determining a position and a motion of the first observer;and determining a position and a motion of the second observer, whereinthe first observer motion parallax view is rendered based upon the firstobserver position and motion, and the second observer motion parallaxview is rendered based upon the second observer position and motion. Thefirst object image is a stereoscopic projection in a space relative tothe display and the first observer motion parallax rendering results ina substantially constant first location projection of the first objectimage in the space relative to the first observer based upon the motionof the first observer and a variable location projection of the firstobject image in the space relative to the second observer based upon themotion of the first observer, and the method further comprises: enablinga first physical object to be observable by the first observer in thespace in addition to the first object image and the second object image;determining the first physical object indicating a first intersectionwith the first location projection of the first object image; andenabling a first process based upon the determining the firstintersection. The second object image is a stereoscopic projection inthe space and the second observer motion parallax rendering results in asubstantially constant second location projection of the second objectimage in the space relative to the second observer based upon the motionof the second observer and a variable location projection of the secondobject image in the space relative to the first observer based upon themotion of the second observer, and the method further comprises:enabling a second physical object to be observable by the secondobserver in the space in addition to the first object image and thesecond object image; determining the second physical object indicating asecond intersection with the second location projection of the secondobject image; and enabling a second process based upon the determiningthe second intersection. The second object image is a stereoscopicprojection in the space and the second observer motion parallaxrendering results in a substantially constant second location projectionof the second object image in the space relative to the second observerbased upon the motion of the second observer and a variable locationprojection of the second object image in the space relative to the firstobserver based upon the motion of the second observer, and the methodfurther comprises: determining the first physical object indicating asecond intersection with the variable location projection of the secondobject image; and enabling a second process based upon the determiningthe second intersection.

In a summary, the disclosure also includes a method comprising:detecting a presence of a first observer and an absence of a secondobserver; enabling a process to render on a display a first observermotion parallax view of an object image based upon the first observer;detecting the presence of the first observer and a presence of thesecond observer able to view the object image rendered on the display;and enabling the process to render on the display the object imagewithout the first observer motion parallax view based upon and thedetecting the presence of the first observer and presence of the secondobserver. The method further comprising: determining a position and amotion of the first observer; determining a position and a motion of thesecond observer; and enabling the process to render on the display acombined first and second observer motion parallax view of the objectimage based upon the determined first observer position and motion andthe determined second observer position and motion. The enabling theprocess to render on the display the object image without the firstobserver motion parallax view based upon and the detecting the presenceof the first observer and presence of the second observer furthercomprises enabling the process to render on the display a secondobserver motion parallax view of the object image based upon the secondobserver and an input signal. The method further comprising: receivingimage data including second observer image data; uniquely identifyingthe second observer based upon the second observer image data; andgenerating the input signal in response to the uniquely identifying thesecond observer.

In a summary, the disclosure also includes a method comprising: enablinga process to render on a display a first observer motion parallax viewof a first extension image based upon a first physical object and afirst observer; and enabling the process to render on the display asecond observer motion parallax view of a second extension image basedupon a second physical object and a second observer. The method furthercomprising: determining a position of the first physical object;determining a position of the second physical object; determining aposition of the first observer; and determining a position of the secondobserver, wherein the rendering on the display of the first observermotion parallax view of the first extension image is based upon thedetermined position of the first physical object and the determinedposition of the first observer; and the rendering on the display of thesecond extension image of the second object with the second observermotion parallax view is based upon the determined position of the secondobject and the determined position of the second observer. The methodfurther comprising: enabling the process to render on the display afirst observer motion parallax view of a third object image based uponthe first observer; and enabling the process to render on the display asecond observer motion parallax view of the third object image basedupon the second observer and an input signal. The third object image isa stereoscopic projection in a space relative to the display and therendering the first observer motion parallax view results in asubstantially constant location projection of the third object image inthe space relative to the first observer based upon the first observerand a variable location projection of the third object image in thespace relative to the second observer based upon the position of thefirst observer, and the method further comprises: enabling a physicalobject to be observable by the second observer in the space in additionto the third object image; determining the physical object indicating anintersection with the variable location projection of the third objectimage; and generating the input signal based upon the determining theintersection.

In a summary, the disclosure also includes an apparatus comprising: atracking system for tracking a position of a first observer and a secondobserver; and a parallax rendering module for rendering a motionparallax view of an object image on a display, the object image viewableby the first observer and the second observer, the motion parallax viewof the object image based upon the first observer and the secondobserver. The apparatus further comprising: a first motion parallaxrendering module for rendering the motion parallax view of the objectimage based on the first observer; and a second motion parallaxrendering module for rendering the motion parallax view of the objectimage based on the second observer, wherein the parallax renderingmodule renders the motion parallax view of the object image based on thefirst observer and renders the motion parallax view of the object imagebased on the second observer and an input signal. The tracking systemgenerates image data including second observer image data, and theapparatus further comprises a facial recognition module for identifyingthe second observer based on the second observer image data and theinput signal is generated based on the identification.

In a summary, the disclosure also includes an abstract summary where adisplay system renders a motion parallax view of object images basedupon multiple observers. The motion parallax view of certain objectimages may be rendered based upon the position and motion of a firstobserver and the motion parallax view of other object images may berendered based upon the position and motion of a second observer. Anobject image rendered with motion parallax based upon a first observermay be intersected by a physical object operated by the second observerat a location perceived by the second observer. Thereafter, the objectimage may be rendered with motion parallax based upon the secondobserver.

II. Augmentation of a Virtual Reality Object

Turning now to augmentation of a virtual reality object. The descriptionfurther addresses a system wherein an observer is wearing a headset ableto detect and interact with object images in a stereoscopic projectionby a display separate from the headset.

FIG. 14 illustrates an example block diagram of a system foraugmentation of a virtual reality object. Headset 1400 may be worn by anobserver able to view a stereoscopic display 1402 included within astereoscopic display system 1404 which may correspond to a motionparallax enabled product such as Learndo 3D, zSpace, and EON Reality'sIbench and Imobile devices or devices 102, 202, 302, 402, 502, 602 and840, of the prior figures, or may correspond to a stereoscopictelevision or movie or other three dimensional projection system. Thestereoscopic display may use any technology able to generate a threedimensional images including technologies known to those familiar withthe art such as a polarized or active shutter technology that utilized aheadset, or autostereoscopy or holograms which requires no headset.Stereoscopic images projected by display 1402 are generated by an imagegenerator 1406 which formats the image for the stereoscopic projectionon the display, the image may be generated from a display database 1408which includes volumetric data indicative of one or more object imagesto be displayed within the stereoscopic image. An example of objectimages stored within display database 1408 includes a flower vase 1410or a wilted daisy flower 1412. The stereoscopic image and object imageare controlled by controller 1414 which may utilize input/output module1416 to communicate signals with other devices 1418. Other devices 1418may include any of a number of devices able to communicate with device1402 such as Blu-ray movie player, other devices similar to device 1402,other internet connected or networked devices, and headset 1400. Theconnection may be any form of wired or wireless interface includingPOTS, HDMI, DVI, Display Port, Ethernet, Internet, Intranet, WiFi,Bluetooth, WiMAX, and LTE.

In order to perceive a stereoscopic image, the right eye 1420 of anobserver receives a right image from display 1402 and the left eye 1421of the observer receives a left image from display 1402. In response, athree dimensional virtual reality object image projected by the displayis perceived by the observer. In the case of an autostereoscopicdisplay, the display 1402 routs left and right images to the left andright eyes of the observer with any of a number of techniques includinga lenticular lens, micro lens, and parallax barrier. Other forms ofstereoscopic projection require a filter at the headset to filter leftand right images for each eye. Such filtering systems known to thosefamiliar with the art include active shutter and polarized filtersystems. Right and left stereoscopic filters, 1424 and 1425, filter theright and left images projected by display 1402 for the right and lefteyes, 1420 and 1421, of the observer. Right and left obstruction filters1428 and 1429 are able to obstruct portions of real and stereoscopicimages to enhance the images viewed by the observer through the headset.Examples of obstruction filters may be found in sunglasses made byDynamic Eye and U.S. Pat. No. 6,559,813.

Right and left cameras 1432 and 1433 are positioned to receiveapproximately the same images received by the right and left eyes 1420and 1421 of the observer. Right and left stereoscopic filters, 1436 and1437, filter the images from the display before being received by thecameras for enabling the camera system to interpret stereoscopicprojections by display 1402. Right and left stereoscopic filter 1424 and1425 for the eyes of the observer may be the same filter element usedfor stereoscopic filters 1436 and 1437 for the cameras of the headset,or may be separate elements. Also note that active shutter and polarizedfilters enable the observer to view both stereoscopic projections ofobjects images projected by the display along with physical objects,such as the hand or finger 1422 of the observer, handheld pointer orwand, or other physical object. In the example of an autostereoscopicdisplay, the stereoscopic filters may be eliminated from the headset.

The operation of the headset is controlled by controller 1440, which mayinclude a computer executing program instructions such as a computerprogram product comprising a storage medium readable by a processingcircuit and storing instructions for execution by the processing circuitfor performing methods described herein. Controller 1440 may alsoinclude a non-transitory computer readable media having a stored set ofinstructions that when executed by a processor or computer cause thedevice to implement at least a portion of the methods or processesdescribed herein. The controller coordinates and controls the processesand methods of the modules.

Object detector 1442 receives the right and left images from right andleft cameras and recognizes objects within the images. The objects mayinclude an object image 1410, such as a stereoscopic projection of avase included within the stereoscopic image projected by the display1402 or a physical object such as the hand or finger 1422 of theobserver. Any recognition technique may be used while remaining withinthe scope of the description. For example, the physical object mayinclude fiducial markers to facilitate recognition and the markers maybe further illuminated with an infrared light source in a techniqueknown to those familiar with the art. In another example, a differentapproach may be used to determine the location of the physical objectrather than stereoscopic cameras 1428 and 1429, such as a systemrealized by the Microsoft Kinect mounted to the headset other optical,radio frequency acoustic receiver or transceiver as well aselectromechanical sensor system associated with the observer, whileremaining within the scope of this description. The obstruction filters1428 and 1429 are shown to be producing an obstruction area 1430 at alocation which is in substantial coincidence with the object image 1412of the wilted daisy, thereby obstructing the view of the wilted daisyfrom the observer, while allowing the observer to view the vase objectimage 1410 projected by stereoscopic display 1402.

Virtual fiducial marks may be included in the object image to facilitaterecognition. For example, yellow fiducial dots rendered on the vase 1410may be recognized by their specific color, or may implement a uniqueright eye, left eye modulation or encoding able to be detected by theheadset object detector. For example, the yellow be rendered as a redmark for the right eye image and a green mark for the left eye image.The observer would perceive a yellow dot while the object detector wouldreceive two different color dots in each image. The difference in colorscan be analyzed as a fiducial marker in a stereoscopic image by theobject detector. Furthermore, the color differential encoding can bemodulated between the eye images to facilitate the fiducial marks in thestereoscopic image. For example a yellow dot as perceived by an observermay actually be encoded as a right eye image which is green fiftypercent of the time and red fifty percent of the time, while the lefteye image is conversely red fifty percent of the time and green fiftypercent of the time. The frequency of modulation may be once per videoframe of the stereoscopic display or less often.

Intersection determiner 1444 determines if the physical object indicatesan intersection with the object image projected by the display 1402. Ifan intersection is detected, then a signal is generated that mayinitiate or modify a process operating in the headset or other device.The modified process in the headset may include sending a signalindicative of the intersection to another device. Communication withanother device may be accomplished through input/output module 1446which may interface with other devices including other device 1418 orstereoscopic display system 1404 through input/output module 1416, whichmay be functionally similar to input/output module 1446 in the headset.The intersection may be determined if the physical object 1442 occupiessubstantially the same location as object image 1410, for example, fromthe perspective similar to the perspective of the observer. Theperspective of the stereoscopic cameras 1432 and 1433 may be similar tothe perspective of the eyes 1420 and 1421 of the observer, andcompensation may be made to account for a predetermined geometricdistance differences between perspectives. For example the cameras maybe offset by a fraction of an inch above the eyes of the observer whenthe headset is worn by the observer. The offset may be compensated formodules of the headset processing cameral images and projectionaugmentation images. The headset may also be able to monitor the eyes ofthe observer to determine the location of the eyes relative to theheadset and based thereupon, determine the offset (and also determinethe direction of gaze of the observer for other interface purposes). Theheadset may also determine that the physical object is indicating anintersection with the object image. For example, the hand of theobserver 1422 is shown pointing to the object image 1410, therebyindicating an intersection with the object image.

The headset also has a headset database 1450 having a volumetric data orother representation of three dimensional objects to be displayed to theobserver. The object is converted to a stereoscopic image by imagegenerator 1452 and projected into the eyes of the observer by right andleft image projectors 1456 and 1457 in a manner known to those familiarwith the art. Any manner of projecting a stereoscopic image from aheadset worn by an observer into the eyes of an observer are includedwithin the scope of this description. FIG. 14 shows headset projectorsprojecting two stereoscopic images, a first stereoscopic imagecorresponding to a stereoscopic extension image 1460 of an arrow orlaser beam extending from a physical object 1422, the finger of theobserver. A second stereoscopic extension image 1470 projected by rightand left projectors 1456 and 1457 is shown as an image of a healthycalla lily flower located in substantial coincidence with obstructionarea 1430.

FIG. 14 illustrates a stereoscopic display system 1404 rendering twoobject images, a vase 1410 and a wilted daisy flower 1412. The headset1400 enables the observer to observe the two object images 1410 and 1412while simultaneously viewing a physical object 1422, which in thisexample is the hand of the observer. Active shutter or polarizedstereoscopic filters 1424 and 1425 allow the observer to perceive athree dimensional view of the object images as well as the threedimensional view of observer's hand, the image of which is passed by thestereoscopic filters in a way that is substantially unmodified from theperspective of the observer. The camera system 1432 and 1433, throughstereoscopic filters 1436 and 1437 enable object detector 1442 todetermine locations of the object images 1410 and 1412 relative to theheadset as well as the location, orientation and position of physicalobject 1422. The headset also has obstruction filters which in thisexample are creating an obstruction area 1430 in substantial coincidencewith wilted daisy object image 1412. Image projectors are also shownprojecting a stereoscopic image 1470 of a healthy calla lily insubstantial coincidence with the obstruction area 1430. Thus, while thestereoscopic display system 1404 is projecting object images of a wilteddaisy in a flower vase, the headset 1400 is augmenting the projectedimage by projecting an image of a health calla lily flower in the flowervase while further obstructing the image of the wilted daisy projectedby the stereoscopic display. Thus, a virtual reality object image isprojected by a stereoscopic display from a first display database, whichis then augmented by the headset with a stereoscopic image from a secondheadset database, the augmented image not being included in the firstdatabase. The headset further augments the observer's view of a physicalobject, by adding a stereoscopic extension image of an arrow or laserbeam emanating from the finger of the observer.

Furthermore, by the observer is pointing to the vase with a physicalobject. The headset is determining that the physical object isindicating an intersection with the object image. A correspondingprocess may be implemented as a result of the intersection. In oneexample of an implemented process, the stereoscopic extension image 1460may be modified by the controller of the headset. If the extension imageis a laser beam having a longer or even infinite perspective length,then its length could be shortened to the length of the vase whenintersecting. Also, pointing to the vase could indicate to thecontroller that a new application should be enabled wherein the wilteddaisy object image is be obstructed and a stereoscopic image of ahealthy calla lily render in its place, thereby augmenting the image ofthe vases in response to the observer pointing to the vases. In anotherexample, information regarding the intersection can be communicated tothe stereoscopic display system 1404 through input/output modules 1416and 1446, in response to which, the stereoscopic display system couldmodify the image of the vase in response thereto. The modification caninclude moving the rendered location of the vase, terminating therendering of the vase or rendering a different object image instead ofthe vase by the stereoscopic display system. In another example, theintersection can be communicated to another device 1418. For example, ifstereoscopic display 1404 was a sign advertising the vase, then theobserver pointing to the vase can be part of a sequence for purchasingthe vase, where the purchase is communicated to another device, such asa server for eBay or Amazon, through the Internet. In this example, theheadset need not communicate with the sign having display 1402 forprojecting the vase, thereby facilitating the transaction without theexpense of a designing and installing sign able to communicate with theheadset.

In another example, stereoscopic image 1460 need not be an extension ofthe physical object and may have a location determined by otherprocesses in the headset. Such an example may be a traveling tennis ballrendered by the headset on a path determined by the controller of theheadset. The headset object detector can also detect the intersectionbetween the stereoscopic image rendered by the headset and an objectimage projected by the stereoscopic display. To build on the example,the object image may be a tennis racket project by the display. Thisintersection is determined to occur at a location where no physicalobject exists, only the virtual image projected by the display and thestereoscopic image rendered by the headset. A response to theintersection can correspond to other intersection responses includingthose described herein. In this example, the response to theintersection may be to modify the path of the tennis ball in response tothe intersection and to generate a sound at the headset similar to thesound of a tennis ball intersecting a tennis racket.

FIG. 15 illustrates an example illustration of a headset worn by anobserver. Headset 1400 has right and left stereoscopic filters 1424,1425 that filter the object images projected by the stereoscopic displayfor the right and left eyes 1420, 1421 of the observer while allowingphysical objects to also be viewed by the observer wearing the headset.Right and left stereoscopic filters 1436, 1437 filter the object imagesrendered by the stereoscopic display for the right and left cameras1432, 1433 of the headset to allow for determination of the location ofthe object images and physical objects and any intersection indication.

FIG. 16A and FIG. 16B illustrate an example of images viewed by anobserver of a stereoscopic projection display without and with theheadset of the disclosure. In FIG. 16A, the observer with a standardstereoscopic headset, such as an active shutter headset or a polarizedheadset, or no headset in the event of a autostereoscopic display, seesa vase 1410 with a wilted daisy flower 1412 projected by the display. InFIG. 16B, the observer wearing headset 1400 sees vase 1410 with ahealthy calla lily 1470. The object image of the wilted flower isreceived by left and right cameras 1432, 1433, detected by objectdetector 1442. As previously discussed, fiducial markers may be imbeddedin the object image 1412 to facilitate detection by the object detector.The fiducial markers may be visible to the observer or hidden in theimage. For example, for one fiducial, if the left most pedal of thedaisy is yellow, it may appear red in the right eye image and green inthe left eye image, wherein the left and right eye images are perceivedas yellow by the observer, but perceived as markers indicative of afiducial by the object detector. Controller 1440 causes left and rightobstruction filters 1428, 1429 to generate an obstruction area 1430 thatobstructs the object image of the wilted daisy 1412 from the observer.Controller 1440 also causes image generator 1452 to generate an imagefrom the headset database 1450 of the health calla lily and render astereoscopic image with left and right image projectors 1456, 1457 ofthe healthy calla lily 1470 in substantial coincidence with theobstruction area 1430. Thus the observer wearing the headset 1400 sees avase 1410 augmented with a healthy calla lily 1470 instead of the wilteddaisy projected by the display. Other examples of augmentation by theheadset need not include the obstruction filter. Furthermore, theaugmentation of the object image may extend beyond the display area ofthe stereoscopic display 1402, thereby providing for an augmentation ofa virtual reality object or image that exceeds the display area of thedisplay projecting the virtual reality object or image. For example, thecalla lily stereoscopic image 1470 may be considerably larger thanwilted daisy image 1412 and extend beyond the boundaries of display1402. This type of extended augmentation is an enhancement over systemsutilizing only display 1402.

FIG. 14 also shows that the headset 1400 is able to simultaneouslyrender augmentation images of both physical objects and object imagesprojected by a stereoscopic display. The stereoscopic image of the callalily flower 1470 augments the object image 1410 projected bystereoscopic display 1402, and stereoscopic extension image 1460augments physical object 1422, the hand or finger of the observer.

FIG. 17A-FIG. 17D illustrate examples of augmentation of a virtualreality object image. In FIG. 17A, a virtual reality object image 1700appears as a candle having a wick which is projected by a stereoscopicdisplay 1402. A physical object 1422 is shown as being augmented by astereoscopic image 1710 rendered by the headset. In this example thestereoscopic image is a flame which is shown as augmenting a physicalobject observable by the observer corresponding to the finger of theobserver. FIG. 17B shows the observer's finger has been moved tointersect with the wick of the candle projected by the display. Theobserver's image remains augmented by the flame rendered by the headset.The object detector 1442 determines the location of the wick relative tothe headset and the finger by processing images from the left and rightcameras 1432, 1433, and intersection determiner 1444 determines theintersection between the wick and the finger and a process is initiatedor modified in response to the intersection.

FIG. 17C illustrates the process initiated by the intersection asrendering a stereoscopic image 1712 of a flame by the headset thataugments the object image 1700 of the candle and wick projected by thedisplay. Note that in the example of FIG. 17C, the stereoscopic image1710 of the flame is retrieved from headset database 1450, and need notbe included in stereoscopic projection system's display database 1408.Similarly, the object image of the candle and wick are retrieved fromthe display database, 1408 and need not be included in the headsetdatabase 1450. Thus, the observer of FIG. 17C perceives a stereoscopicimage of a candle with a flame when neither the headset nor thestereoscopic projection system has a database with the complete image ofa candle and a flame.

FIG. 17D illustrates an alternate example. The intersection detected inFIG. 17B results in a process that sends a signal from the headset 1400to the stereoscopic projection system 1404 indicative of theintersection. In this example, in response to receiving the intersectionsignal, an object image 1720 of a flame is retrieved from the displaydatabase 1408 and projected in coincidence with the wick of the candleof the object image 1700 by stereoscopic display 1402. The headsetcontinues to render a stereoscopic image 1710 of a flame on the physicalobject 1422, the finger of the observer. Note that since flame 1710 isretrieved from the headset database 1450 and flame 1720 is retrievedfrom display database 1408, they may not have an identical appearance.Indeed flame 1710 and flame 1720 are shown as having differentappearances. In another example the headset 1400 and display system 1404may exchange data through input/output modules 1416, 1446 related toflame images 1710 and 1720 to provide a uniform appearance to theobserver.

FIG. 18 illustrates an example of an intersection between an objectimage projected by a stereoscopic display and a stereoscopic imagerendered by a headset. Stereoscopic display 1402 is projecting objectimage 1800 shown as a handheld tennis racket. Headset 1400 is renderinga stereoscopic image 1810, 1812 of a tennis ball in flight. In thisexample, right and left cameras 1432, 1433 receive right and left imagesof the object image, which are processed to determine the location ofthe object image relative to the headset. The controller 1440 causes animage of a tennis ball from headset database 1450 to be processed byimage generator 1452 and rendered by right and left image projectors1456, 1457 to move along a path as shown by 1810 and 1812. Theintersection determiner 1444 determines the location of the tennisracket object image relative to the headset and determines if thelocation of the stereoscopic tennis ball rendered by the headsetintersects with the determined location of the object image. If thetennis ball intersects the tennis racket, as shown by stereoscopic image1812 being in substantial coincidence with object image 1800, and inresponse a process is enabled. In this example, the headset enables aprocess that generates a sound 1820 in response to the intersection. Inthis example, the headset includes one or more speakers allowing theobserver to hear sounds generated by the headset. In this example, thesound generated by the headset in response to the intersection is a“smack” sound corresponding to the sound created by an intersectionbetween a real physical tennis ball impacting a real physical tennisracket.

FIG. 19 illustrates a representative flow diagram for determining anintersection with an object image. In step 1902 a left stereoscopicprojection image is filtered for a left camera of the headset. In step1904 a right stereoscopic projection image is filtered for a rightcamera of the headset. The filter of step 1902 and 1904 may be based onactive shutter or polarization technology at the headset or based onautostereoscopic technology at the display. Step 1906 determines alocation of the object image in the projected stereoscopic imagerelative to the headset. Step 1908 determines a position, locationand/or orientation of a physical object. The determination of step 1908may be made by processing the images received by the cameras, or otherapproaches for making such a determination regarding a physical object.Step 1910 (optionally) renders a stereoscopic extension image of thephysical object at the headset. In step 1912 the determination is madewhether or not the stereoscopic extension image or the physical objectindicates and intersection with the object image. If so, then step 1914enables a process based upon determining the intersection. The processmay include modifying the stereoscopic extension image, launching anapplication at the headset or other connected device, generating analert (audio, visual, tactile or otherwise), generating an output signalfor reception by another device, and modifying the stereoscopicprojection image projected by the display separate from the headset.Note that in this process, in an example where the stereoscopicextension image of step 1910 image is not required, then projectors 1456and 1457 are not required in the headset in order to determine anintersection between the physical object and the object image.

FIG. 20 illustrates a representative flow diagram for augmenting anobject image projected by a stereoscopic display with a stereoscopicimage rendered by a headset. In step 2002 a left stereoscopic projectionimage is filtered for a left camera of the headset. In step 2004 a rightstereoscopic projection image is filtered for a right camera of theheadset. The filter of step 2002 and 2004 may be based on active shutteror polarization technology at the headset or based on autostereoscopictechnology at the display. Step 2006 determines a location of the objectimage in the projected stereoscopic image relative to the headset. Step2008 renders the stereoscopic image, or stereoscopic augmentation imageat the headset based upon the determined location of the object image.Note that in this process, a physical object and its intersection withthe object image need not be determined.

FIG. 21 illustrates a representative diagram of multiple observersobserving and intersecting an object image projected by a stereoscopicdisplay. Display 1402 is rendering a stereoscopic object image 2105while being simultaneously observed by three observers 2110, 2120, and2130. The object image is perceived at location 2115 by observer 2110wearing headset 2112, and is perceived at location 2125 by observer 2120wearing headset 2122, and is perceived at location 2135 by observer 2130wearing headset 2132. Observer 2110 is shown intersection object image2115 with physical object 2117 corresponding to the finger of theobserver. In determining the intersection, the headset 2112 determinesthe location of the object image 2115 relative to the headset and thelocation of the physical object 2117. Observer 2120 is shown withphysical object 2127 which may be a wand or pointer held by theobserver. The headset 2122 determines the location, orientationand/position of the physical object 2127 and renders a stereoscopicextension image 2129 based on the physical object. In this example thestereoscopic extension image appears as a laser beam emanating from thewand or pointer. Since the physical object 2127 is not indicating anintersection with object image 2125, no intersection is detected, andstereoscopic extension image 2129 is rendered as longer than thedistance to the object image 2125. Observer 2130 is shown with physicalobject 2137 which may be a wand or pointer held by the observer. Theheadset determines the location, orientation and/position of thephysical object and renders a stereoscopic extension image 2139 based onthe physical object. In this example the stereoscopic extension imageappears as a laser beam emanating from the wand or pointer. Since thephysical object 2137 is indicating an intersection with object image2135, an intersection is detected, and stereoscopic extension image 2129is rendered with a length terminating at the object image 2135. Thisgives the observer the perception that laser beam 2139 is hitting objectimage 2135.

The difference in length between stereoscopic extension image 2129 andstereoscopic extension image 2139 shows another example of how astereoscopic image rendered by the headset can be modified based upondetermining an intersection. Stereoscopic image 2129 has a first length,and in response to a determined intersection a process is enabled thatadjusts the length of the stereoscopic image 2139 to correspond to theintersection with the object image. Thus, a stereoscopic image renderedby the headset is modified in response to an intersection with an objectimage projected by a stereoscopic display. In another example, headset2112 detects an intersection between a physical object 2117 and anobject image 2115, and headset 2132 detects an intersection between aphysical object 2137 indicating and intersection with an object image2135 (stereoscopic extension image 2139 notwithstanding). Furthermore,in another example where stereoscopic image 2139 was rendered by headset2132 independent of any physical object 2123, then headset 2132 woulddetect an intersection between a stereoscopic image 2139 rendered by theheadset and an object image 2135 projected by the display.

Furthermore, observers 2110, 2120 and 2130 are able to independentlyinterface with object image 2105. The laser beam 2129 is only visible toobserver 2120 because it is rendered by headset 2122 and is notprojected by display 1402. This is contrasted with laser beam 372 ofFIG. 3A which is visible to both observers 300 and 350 even though laserbeam 372 does not appear to be emanating from physical object toobserver 300. In one example, stereoscopic display 1402 is a threedimensional display at a movie theater and the object image is a buttonthat says “push for popcorn”. Observers 2110 and 2130 are indicating adesire for popcorn by indicating an intersection with the object image.In response, headsets 2112 and 2130 communicate the desire for popcornby enabling a process that communicates a signal indicative of the orderfor popcorn to a wireless server within the movie theater. Further,object image 2105 could be one of several object images simultaneouslyprojected by display 1402 allowing theater patrons to order concessionsuch as popcorn, soft drinks and candy. Many other examples ofapplications of the system of this description may be realized whileremaining within the scope of the description. Furthermore, in this andother examples, an object image, such as object image 2105, may belocated in the plane of display 1402, thereby appearing as a twodimensional object displayed by a conventional non-stereoscopic twodimensional display, in which case display 1402 need not be astereoscopic display.

FIG. 22 illustrates and example of a flow diagram for determiningintersections with an object image projected by a stereoscopic displayand determined to be located in multiple locations relative to each ofthe multiple headsets. Step 2202 determines a location of an objectimage in stereoscopic projection at a first headset. Step 2204 thendetermines the orientation, position and/or location of a first physicalobject. Step 2206 then (optionally) renders a first stereoscopicextension image of the first physical object at the first headset. Ifthe first physical object and/or the first stereoscopic extension imageindicates an intersection with the object image at the first location atstep 2208, then step 2210 enables a first process. The first process mayinclude modifying the stereoscopic extension image, launching anapplication, generating an alert, generating an output signal, and/ormodifying the stereoscopic extension image. Step 2212 determines asecond location of the object image in stereoscopic projection at asecond headset. Step 2214 then determines the orientation, positionand/or location of a second physical object. Step 2216 then (optionally)renders a second stereoscopic extension image of the second physicalobject at the second headset. If the second physical object and/or thesecond stereoscopic extension image indicates an intersection with theobject image at the second location at step 2218, then step 2220 enablesa second process. The second process may include modifying thestereoscopic extension image, launching an application, generating analert, generating an output signal, and/or modifying the stereoscopicextension image. Thus, FIG. 22 shows a process where multiple observerswearing multiple headsets can independently interface with a commonobject image projected by a single stereoscopic display. The singlestereoscopic display need not monitor physical objects associated witheach of the multiple observers. In this example, hundreds or eventhousands of observers in a theater or large conference room or otherfacility can independently interface with one or more object imagesprojected by a single stereoscopic display.

In a brief summary example, a method comprises receiving, at a headset,an object image projected by a stereoscopic display separate from theheadset, the headset configured to enable the object image and aphysical object to be simultaneously observed by an observer wearing theheadset; determining a perceived location of the object image relativeto the headset; determining the physical object indicating anintersection with the determined location of the object image; andenabling a process based upon the determining of the intersection. Themethod further comprising: receiving, at a second headset, the objectimage projected by the stereoscopic display separate from the secondheadset, the second headset configured to enable the object image and asecond physical object to be simultaneously observed by a secondobserver wearing the second headset; determining a second perceivedlocation of the object image relative to the second headset; anddetermining the second physical object indicating a second intersectionwith the determined second location of the object image; and enabling asecond process based upon the determining of the second intersection.The process communicates a signal indicative of the intersection to adevice controlling the projection of the object image upon the display,and the second process communicates a second signal indicative of thesecond intersection to the device controlling the projection of theobject image upon the display. The method further comprising: rendering,at the headset, a stereoscopic extension image based upon the physicalobject, wherein the process modifies the stereoscopic extension imagebased upon the determined intersection. The process communicates asignal indicative of the intersection to a device controlling theprojection of the object image by the stereoscopic display. The processmodifies the object image. The headset includes a camera systemconfigured to receive a camera image including the object image and animage of the physical object and the determining the location of theobject image is based upon the camera image, and the determining thephysical object indicating the intersection is based upon the cameraimage. The camera system includes a left camera and a right camera andthe object image includes a left image combined with a right image, andthe method further includes: filtering, at the headset, the left imagefrom the object image for both the left camera and a left eye of theobserver; and filtering, at the headset, the right image from the objectimage for both the right camera and a right eye of the observer. Thecamera system includes a left camera and a right camera and the objectimage includes a left image directed towards a left eye of the observerand a right image directed towards a right eye of the observer, and themethod further includes: receiving, at the headset, the left image withthe left camera; and receiving, at the headset, the right image with theright camera.

In another brief summary example, a method comprises: receiving, at aheadset, an object image projected by a stereoscopic display separatefrom the headset; determining a perceived location of the object imagerelative to the headset; and rendering, at the headset, a stereoscopicimage based upon the location of the object image, whereby thestereoscopic image augments the object image. The object image isprojected based upon a first database and the stereoscopic image isrendered based upon a second database, at least a portion of which isnot included in the first database. The stereoscopic image at leastpartially augments the object image based upon the portion of the seconddatabase not included in the first database. The method furthercomprising the step of generating, at the headset, an obstruction areaadapted to block at least a portion an image projected by thestereoscopic display. The method further comprising the step ofgenerating, at the headset, an obstruction area adapted to block atleast a portion of the object image, wherein the stereoscopic imagerendered by the headset is at least partially rendered within theobstruction area. The method further comprising: receiving, at theheadset, a physical object image of a physical object; determining alocation of the physical object relative to the headset; and rendering,at the headset, a second stereoscopic image based upon the location ofthe physical object. The stereoscopic image at least partially augmentsthe object image, and the second stereoscopic image at least partiallyaugments the physical object. The physical object, the stereoscopicdisplay projecting the object image, the object image, the stereoscopicimage, and the second stereoscopic image are able to be simultaneouslyviewed by an observer wearing the headset.

In another brief summary example, a method comprises: receiving, at aheadset, an stereoscopic projection having an object image, thestereoscopic projection rendered upon a display separate from theheadset; determining a location of the object image relative to theheadset; rendering, at the headset, a stereoscopic image; determiningthe stereoscopic image indicating an intersection with the determinedlocation of the object image; and enabling a process based upon thedetermining of the intersection. The enabling the process includesmodifying the stereoscopic image rendered at the headset. The enablingthe process includes transmitting a signal indicative of theintersection to a device external to the headset.

In another brief summary example, a headset comprises: a right camera; aright image filter configured to filter a right eye image of astereoscopic projection for reception by the right camera and forviewing by a right eye of an observer wearing the headset; a leftcamera; and a left image filter configured to filter a left eye image ofthe stereoscopic projection for reception by the left camera and forviewing by a left eye of the observer wearing the headset. The rightimage filter includes: a first right image filter configured to filterthe right eye image of the stereoscopic projection for viewing by theright eye of the observer; and a second right image filter configured tofilter the right eye image of the stereoscopic projection for receptionby the right camera, and the left image filter includes: a first leftimage filter configured to filter the left eye image of the stereoscopicprojection for viewing by the left eye of the observer; and a secondleft image filter configured to filter the left eye image of thestereoscopic projection for reception by the left camera. The left imagefilter and the right image filter include at least one of a polarizedfilter and an active shutter filter. The stereoscopic projectionincludes an object image and the headset further comprises: an objectdetector coupled to the right camera and the left camera configured todetermine a location of the object image relative to the headset; and anintersection determiner configured to determine a physical objectindicating an intersection with the determined location of the objectimage. The right image filter is further configured to enable viewing,through the right image filter, a physical object by the right eye ofthe observer wearing the headset, and the left image filter is furtherconfigured to enable viewing, through the left image filter, thephysical object by the left eye of the observer wearing the headset. Thestereoscopic projection includes an object image, the right camerafurther receives a right perspective image of the physical object, theleft camera further receives a left perspective image of the physicalobject, and the headset further comprises: an object detector coupled tothe right camera and the left camera configured to determine a locationof the physical object based upon the right perspective image and theleft perspective image, the object detector further configured todetermine a location of the object image relative to the head set; andan intersection determiner configured to determine the physical objectindicating an intersection with the determined location of the objectimage.

In a summary, the disclosure also includes an abstract summary where aheadset renders stereoscopic images that augment either physical objectviewed through the headset, or virtual objects projected by astereoscopic display separate from the headset, or both. The headsetincludes a system for locating both physical objects and object imageswithin a stereographic projection. The headset also includes a projectorfor projecting a stereographic image capable of augmenting the physicalobject, the object image projected by the stereographic display, orboth. The headset includes obstruction filter for obstructing portion ofthe stereographic projection for enhanced augmentation of imageprojected by the stereographic display. The headset includes anintersection determiner for determining if an intersection has occurredbetween an object image projected by the stereographic display andeither a physical object, a stereographic image rendered by the headset,or both. In response to the intersection determined at the headset, aprocess may be enabled in either the headset, a device controllingstereographic display, or another device external to the headset.

III. Proprioceptive User Interface

Turning now to the proprioceptive user interface, FIG. 23 shows a sideview of an observer interfacing with a proprioceptive user interface.The observer 2300 is show in a seated position and at times interfaceswith a system able to realize the features of a proprioceptive userinterface. The observer has body parts such as a head 2302, a nose 2304,a shoulder 2306 as well as a neck and an eye. In the proprioceptive userinterface, interface locations are positioned relative a body part.Circular interface location 2314 is shown to be forward and up relativeto the nose 2304 of the observer and square interface location 2316 isshown to be forward relative to the shoulder 2306 of the observer. Leftor right locations relative to the observer are not shown in the sideview of FIG. 23. The interface location moves as the body part moves.The interface location is associated with a function or processimplemented by the proprioceptive user interface. The function orprocess is implemented when a physical object either intersects theinterface location or indicates an intersection with the interfacelocation, such as with an extension vector or a stereoscopic extensionimage. Since the interface location relative to a body part remainssubstantially constant, the observer becomes familiar with the interfacelocation relative to the body part, as an extension of the body part,and may activate the function without requiring a display system torender an icon or interface object at the interface location. This freesthe observer's field of vision from unnecessary clutter and reducesrequirements of a display system associated with the proprioceptive userinterface to render interface icons for use with the interface. Forexample, eliminating a requirement to render icons or interface objectson a system with a stereoscopic display terminal, such as terminal 102of FIG. 1 allows for interface locations to remain functional eventhough their location may be beyond the limited projection space of thestereoscopic display. Also, for interface locations within the limitedprojection space of the terminal, since an icon or object image need notbe rendered, more display space may be available for other content beingrendered. For example, a stereoscopic rendering of a simulation of anopen heart surgery being practiced by a physician observer need not beobscured by icons or object images identifying various interfacelocations. For example an icons for scalpels or clamps need notdisplayed if their interface location is known by the physician usingnatural proprioception. Since the observer has become familiar with theinterface locations through a human's natural proprioception abilities,their functionality or associated process may implemented by a physicalobject, such as the finger of the observer or wand held by the observer,indicating an intersection with the object image, even though no icon orobject image is rendered at the interface location. The proprioceptiveuser interface enables this because the interface location is known bythe observer to be relative to a body part of the observer.

The body part may be further identified with an element or a referenceelement. In one example, an element or reference element is a body partitself such as a head, nose, eye, neck or shoulder of the observer. Inanother example the element or reference element may be attached to orassociated with a body part. Such an element may include one or morefiducial marks facilitating identification of a body part, such as a setof glasses worn by the observer with one or more reflective or radiativeelements, such as the glasses of the zSpace terminal, to facilitate adetermination of the body part by an imaging system associated with thesystem implementing the proprioceptive user interface. In such anexample the proprioception interface locations would be located relativeto the head of the observer. Also note that since relative positions ofthe eyes, and nose do not change relative to the head of the observer,such interface locations may also be considered relative to the eyes ornose (or other body parts included within the head) of the observer.

In another example, the reference element may be removably affixed to abody part of an observer, such as a headset or an article of clothinghaving a fiducial mark and worn by the observer. A physician wearing asurgeon garb may have an interface location in front of a shoulderindicating selection of a scalpel. The shoulder portion of the articleof clothing may have a fiducial mark identifying the shoulder. If thephysician is dressed in street clothes, then the interface location infront of the shoulder may control audio characteristics of an audiosystem to which the physician wishes to interface. Thus, thefunctionality of an interface location may be changed based upongarments or other items worn by the observer.

In another example, the reference element may be a head worn deviceincluding the imaging system. The imaging system determines the locationof the interface system relative to the body part, and if the body partis the head of the observer and the headset is worn on the head of theobserver, the interface location may be determined relative to theimaging system itself. The imaging system may further determine thephysical object indicating an intersection with the interface location.The head worn device may also include a display system. Devices such asheadsets are described by the Google Glass or the Atheer headset andhave both imaging and display systems included within a head worndevice. In these examples, the interface location 2314, 2316 may have adetermined physical relationship with the reference element by beingfixed is space relative to the reference element, and moving with themovement of the reference element.

The observer 2300 is shown reaching towards interface location 2316 withtheir left arm. Note that no icon or object image needs to be renderedat location 2316 because proprioception allows the observer to know theinterface location is a fixed distance in front of the left shoulder2306 of the observer. If the function or process enabled as a result ofa physical object indicating an intersection with the interface locationdoes not require a display system, then no display system is required.For example if the intersection were to enable a process in a musicplayer to skip to the next song, then only a system for determining theinterface location and corresponding intersection is required. Such asystem may include an imaging system or other such system known to thosefamiliar with the art. One advantage being that a display system neednot be a requirement.

While interface locations may be placed relative to any body part, oreven in coincidence with a body part, placing interface locationsrelative to the head or shoulders has a possible advantage of freeingthe arms, hands, fingers, legs, feet and toes of the observer for otherfunctions, such as walking, eating or other activities involving theseextremities without activating the proprioceptive user interface.Furthermore, placing the interface locations a short distance away fromthe head and shoulder of the observer has a possible advantage ofallowing for activities including personal hygiene like blowing of thenose of the observer without activating the proprioceptive userinterface. In such an example, the interface locations may be placedwithin an arm's length of the observer or between two centimeters andfifty centimeters of the eye of the observer. Another potentialadvantage includes placing the interface locations so they appear withinthe peripheral vision of the observer to enable visual location of thephysical object, such as a finger of the observer, intersection aperceived location of the interface location. For example an interfacelocation two centimeters to the left and two centimeters forward of theleft eye of the observer may be within the peripheral vision of theobserver, enabling the observer to see a finger of the observer's lefthand intersecting the interface location without occupying the primaryfield of vision directly in front of the observer.

Furthermore, interface locations in close proximity with an observer'sface shoulders are generally considered by many social norms to bewithin the personal space of the observer, a space which is generallynot accessed by others. Thus the proprioceptive user interface may beconsidered a very personal—personal user interface—not only because itneed not be displayed, but further because it is located in a space thatgenerally not accessed persons other than the user.

FIG. 24A and FIG. 24B show a proprioceptive user interface havinginterface locations relative to various body parts that move with thevarious body parts. FIG. 24A shows an observer 2400 having body partsincluding a nose, 2404, a left shoulder 2406, a right shoulder 2407, aleft eye 2408 and a right eye 2409. The head of the observer is facingforward. Interface location 2414 is shown relative to the nose 2404 ofthe observer as being forward and to the left, and interface location2415 is shown relative to the observer as being farther forward and lessfar to the left as interface location 2414. The left shoulder 2406 isshown as having an interface location 2416 that is forward and to theright relative to the left shoulder, and right shoulder 2407 is shown ashaving an interface location 2417 that is forward relative to the rightshoulder.

FIG. 24B shows the head of observer being turned to the left relative tothe shoulders of the observer. Note that interface locations 2416 and2417 remain constant relative to FIG. 24A because the shoulders 2406 and24047 of the observer remain in the same position. However, interfacelocations 2414 and 2415 have rotated with the head or nose 2404 of theobserver. Even though interface locations 2414 and 2415 have movedrelative to interface locations 2416 and 2417, the observer may useproprioception to intersect the interface locations without the aid ofan augmentation of the interface location by a display system renderingan icon or interface image at the interface location, because theinterface location relative to their associated body parts remains eventhough the observer may be regularly changing positions. The observeruses natural proprioceptive abilities to know the current location ofthe observer body parts and extends the knowledge to know the relativelocations of the interface locations without the aid of augmentation ofthe interface location.

FIG. 25A, FIG. 25B and FIG. 25C illustrate examples of limitedprojection space and imaging space of various devices able to implementa proprioceptive user interface. FIG. 25A shows an example of astereoscopic display terminal 2502 similar to terminal 202 of FIG. 2 forimplementing a proprioceptive user interface. Based upon the position ofobserver 2500, the stereoscopic display has a limited projection space2504 for rendering stereoscopic images. The limited display space isbased on the size of the display and the volume in which stereoscopicimages can be generated. The volume between the observer and the displayis limited to the space in which both eyes of the observer have a viewof the display. Stereoscopic images rendered by the stereoscopic displayhave many uses, some of which have been described herein, and furtheruses may include augmenting interface locations of the proprioceptiveuser interface with an icon or object image if the interface location iswithin the limited projection space of the stereoscopic display. Theobject image may nevertheless not be rendered at the interface locationeven though the interface image is within the limited projection space:one reason for not rendering the object image is that the observer isfamiliar with the location of the interface location and does notrequire the object image to be rendered. This leave more display spaceor display volume or area for the rendering of other content by thedisplay system, while maintaining the functionality of the interfacelocation.

On the other hand, the object image may be rendered at the interfacelocation in order, for example, to facilitate learning or refreshingknowledge of interface locations for the observer. The object image maynot be rendered if the observer is familiar with the interface locationrelative to an observer's body part using proprioception. The imagingsystem has an imaging space 2506 which is shown to be larger than thelimited projection space. This allows for determination of the referenceelement associated with the body part, the interface location as well asdetermination of a physical object indicating an intersection with theinterface location by the imaging system whether or not the interfacelocation is beyond the limited display area 2504. The imaging system mayinclude a camera array, either infrared, visible or ultraviolet, fordetermining the location of the body part of the observer or thereference element is itself the body part to which the interfacelocation in relative to. Other approaches to locating interfacelocations relative to body parts and/or reference elements associatedwith body parts are considered to be within the scope of thisdescription.

FIG. 25B shows an example of an observer 2510 wearing a head mounteddevice 2512 such as head mounted device 1400 of FIG. 14. The headsetdisplay system has a limited projection space 2514 as defined by thedisplay system of the headset and a larger image space 2516 as definedby the imaging system of the headset. The larger imaging space shown inFIG. 25B may be realized by additional cameras directed towards theshoulders of the observer. The larger imaging space allows the headmounted device to determine the location of a body part of the observer,such as a shoulder of an observer. Since the device is worn on the head,the head worn device itself may be viewed and a reference elementassociated with a body part of the observer, wherein the body part isthe head of the observer. The head worn device moves as the head of theobserver moves and the interface locations move with the headset. Thusinterface locations relative to the head of the observer may also beconsidered as relative to the headset when the observer is wearing theheadset having a substantially constant location on the head of theobserver. Similar to the terminal of FIG. 25A, an interface locationwithin the limited display space 2514 of the headset may be augmentedwith an object image rendered by the headset, or need not be augmentedbased perhaps upon the observer's familiarity with the proprioceptiveuser interface.

FIG. 25C shows an example of an observer 2520 holding a handheld orportable device, such as the device of FIG. 26, which may be acellphone, smart phone, tablet, handheld gaming device or other portabledevice having a proprioceptive user interface. The device has a limiteddisplay space 2524 which is shown extending to the rear of the device,and a rear imaging space 2526 for determining an interface locationrelative to a reference element associated with a body part of theobserver. In this example, if an interface location is within thelimited display space 2524, then it may or may not be augmented with arendering by device, 2522. The portable device has the further advantagein that it may itself be the physical object that indicates anintersection with the interface location. Also, the device may include afront imaging space 2528 for determining if other physical objects, suchas the other hand of the observer, are indicating an intersection withthe interface location.

In other examples, the limited projection space may extend forward andrearward, and the display may be a stereoscopic display (3D) renderingthree dimensional objects or may be two dimensional display rendering athree dimension depth perspective (2.5D) of an object image augmentingthe interface location.

FIG. 26 shows an example of a basic block diagram of the portable devicehaving a proprioceptive user interface. Portable device 2600 includes arear imaging system 2602, which may include one or more cameras, and adisplay. The rear imaging system has a rear imaging space as shown byspace 2526 of FIG. 25C. Cellphones, tablets and laptops commonly havedisplays and rear cameras for facilitating a number of operationsincluding video conferencing and self-portrait photographs. Thearrangement allows the portable device to receive an image of anobserver observing the display. The display may be a stereoscopicdisplay or a conventional two dimensional display. A front imagingsystem 2606 receives images in front of the device and may be used forreceiving images of physical objects indicating an intersection with aninterface location or general purpose photography available on most cellphones. The front imaging system may include one or more cameras. Thedevice also includes a controller system 2608 which may have componentssimilar to the components of the block diagrams of FIG. 2 and FIG. 14for determining interface locations, body part locations, physicalobjects, intersections and rendering images. The device may include anobserver locator for determining the location of the observer as well asreference elements associated with body parts of the observer based uponimages captured by the imaging system, an interface location determinerto determine interface locations based upon the reference elementsidentified from images of the imaging system. It may also include anobject image rendering determiner able to determine if an interfacelocations occurs within the limited projection space of the displaysystem.

FIG. 27A and FIG. 27B show an example of an operation of aproprioceptive user interface for an observer wearing a head mounteddevice such as a headset. Observer 2400 is wearing a headset 2512,similar to headset 1400 of FIG. 14. Headset 2512 is implementing theproprioceptive user interface of FIG. 24. Interface locations 2414 and2415 are located relative to the headset, which is worn by the observeron the nose 2404 of the observer and is thus relative to a body part ofthe observer, which in this example is the nose of the observer.Interface location 2417 is located forward relative to the rightshoulder 2407 of the observer. Interface location 2416 is locatedforward and to the left of the left shoulder 2406 of the observer.Interface locations 2414, 2415, and 2417 are located within the limitedprojection space 2514 of the headset display system and are shown asfilled circles and rectangles. Interface location 2416 is beyond thelimited display projection space 2514 and is shown as an unfilledrectangle. Interface locations 2414, 2415, and 2417 may or may not beaugmented with an object image rendered by the display system of theheadset because they are located within the limited projection space2514. The augmentation may be dependent upon the observer's familiaritywith the proprioceptive user interface. The observer may prefer theaugmentation while learning the proprioceptive user interface andthereafter manually disable the augmentation because the observer hasbecome familiar with the interface locations relative to the body partsof the observer. In another example, the augmentation may be temporarilyenabled based upon the physical object either indicating an intersectionwith the interface location or approaching an intersection with theinterface location. Thus, as a physical object gets close to thelocation of the interface image, augmentation of the interface locationmay be enabled and an object image rendered.

FIG. 27B shows the observer's head rotated to the left. Interfacelocation 2417 is no longer within the limited projection space 2514 ofthe headset and is shown as an unfilled rectangle. Interface location2416 is now within the limited projection space 2614 of the headset andis shown as a filed rectangle. Interface locations 2414 and 2415 moverelative to the nose of the observer and thus rotate with the headset.These figures show that interface locations located relative to the headof the observer may remain in the limited projection space of anobserver wearing a headset, while interface locations located relativeto other body parts may be viewed by the head movements (or movements ofthe shoulders) of the observer that bring the interface location withinthe limited projection space of the headset.

As previously described the headset includes an imaging system capableof locating the reference body parts, such as shoulders 2406 and 2407,and is able to detect a physical object indicating an intersection withthe interface location. In one example, the physical object may be afinger of the observer, or a wand held by an observer. The physicalobject may indicate an intersection with the interface location by, forexample, directly intersecting the interface location or by pointingtowards the intersection and the line of the point may be augmented witha laser beam or other end effect. Other methods of indicating anintersection are within the scope of this description.

FIG. 28A and FIG. 28B show an example of an operation of aproprioceptive user interface with an observer viewing a stereoscopicterminal. In this example the observer of FIG. 27 may have removedheadset 2512 and is now observing stereoscopic terminal 2502. Note thatinterface locations 2414-2417 remain constant relative to the referenceelements associated with the various body parts of the observer eventhough the observer of FIG. 27 is wearing a headset and the observer ofFIG. 28 is observing a stereoscopic terminal. Terminal 2505 is similarto the block diagram of the terminal of FIG. 8 and is operating with aproprioceptive user interface. Interface locations 2414 and 2415 areshown to be within the limited projection space 2504 of the stereoscopicterminal in both FIG. 28A and FIG. 28B. Thus object images may berendered by the terminal to augment these interface locations. Interfacelocation 2417 is shown to be beyond the limited projection space 2504 inFIG. 28A. In FIG. 28B the observer has rotated their right shoulder 2407to move the interface location 2417 to occur within the limitedprojection space 2504. Moving in such a way as to locate interfacelocation 2417 within limited projection space 2504 may enable theobserver to discover or refresh their familiarity with the locationand/or functionality of interface location 2417 as it is locatedrelative to the right shoulder 2407. Interface location 2416 in notwithin the limited projection space 2504 in either FIG. 28A or FIG. 28B.However, the imaging space of the imaging system of terminal 2502enables the terminal to determine the locations of interface locations2414-2417 whether or not the interface locations are within the limitedprojection space 2504. Thus, the observer may cause a physical object toindicate an intersection with any of the interface locations whether ornot they are augmented with object images and whether or not they arewithin the limited projection space of the terminal.

FIG. 29 shows a portable device having a display system and operating aproprioceptive user interface. Observer 2400 is holding a portabledevice 2522 which may be a cell phone, tablet, personal computer,portable gaming system or other device operating a proprioceptive userinterface and may have a block diagram similar to the device of FIG. 26.The observer of FIG. 29 may be the same observer of FIG. 27 and FIG. 28,but in FIG. 29, the observer may have set aside the headset and terminaland in this example is using a cell phone to operate the proprioceptiveuser interface. Interface locations 2414-2417 remain constant relativeto their reference element associated with a respective body part of theobserver. The portable device has a display and is located between theobserver and interface locations 2415 and 2417. The portable device hasa camera system facing the observer and is able to determine observerbody part locations from which interface locations are relativelylocated. Interface locations 2415 and 2417 are located within thelimited display space of the display and thus may be augmented byrendering interface objects appearing to the observer to be located atthe interface locations, the augmentation occurring by images renderedon the display of the portable device. The display of the portabledevice may be a stereoscopic display, or may be a two dimensionaldisplay rendering interface objects in 2.5D to give appropriate depthperception to the object images.

FIG. 30 shows object images rendered on the display of the portabledevice of FIG. 29 that augment the interface locations within thelimited display area 2524 of the portable device. The display 2406 showsinterface location 2417 being augmented by rendering a rectangular imageappearing to occur at a location corresponding the interface locationfrom the perspective of the observer. Similarly since interface location2415 is behind interface location 2417 from the perspective of theobserver, the object image augmenting interface location 2415 isrendered to appear behind the object image rendered to augment interfacelocation 2417. The observer may move the location of the portable deviceto enable augmenting of other interface locations by moving the limiteddisplay space to include other interface locations. For example,rotating or moving the portable device to the left would allow foraugmentation of interface location 2414, while further rotation ormoving to the left would allow for augmentation of interface location2416.

FIG. 31 shows an example of the portable device of FIG. 29 operating aproprioceptive user interface that is rendering an object imageaugmenting an interface location while acting as a physical object forindicating an intersection with the interface location located relativeto a body part of an observer. With reference to FIG. 29, the observerhas moved the portable device forward in order to cause the portabledevice to intersect with interface location 2417, which is locatedrelative to the right shoulder 2407 of the observer. In response to theintersection, a process based upon the intersection is enabled. Forexample, interface location 2417 could indicate a muting or unmuting ofaudio produced by the portable device, and the object image augmentingthe interface location could appear as a speaker when the portabledevice is muted and as a speaker with cross or circle with a slash overthe speaker when the portable device is unmuted. Thus, if the portabledevice is muted, the intersection of the portable device with interfacelocation 2417 would enable a process to unmute the portable deviceaudio, and if the portable device is unmuted, the intersection of theportable device with interface location 2417 enable a process to mutethe portable device audio.

In another example, the portable device could indicate the intersectionwith interface location 2417 without actually intersecting interfacelocation 2417. If the display of FIG. 30 included a touch screen, thenthe intersection could be indicated by the device receiving a touchinput on the touch screen corresponding to the area on the display ofFIG. 30 used for rendering an object image for augmenting interfacelocation 2417. Alternately, a track pad, track ball, mouse or other typeof user input receiver may be used in place of a touch screen forindicating an intersection with an interface location.

FIG. 32 shows an example of the portable device of FIG. 29 operating aproprioceptive user interface that is not necessarily rendering anobject image augmenting an interface location while acting as a physicalobject for indicating an intersection with the interface locationlocated relative to a body part of an observer. FIG. 32 is differentfrom FIG. 31 in that interface location 2417 does not need to beaugmented with a rendered object image because the observer is notviewing the portable device. Another reason that the object image neednot be rendered includes the observer not requiring augmentation do tothe familiarity with the interface location. Even though the objectimage is not augmented and/or the observer is not looking towards theinterface location, the portable device is able to determine thelocation of the right shoulder 2407 of the observer, the location ofinterface location 2417 relative to the right shoulder and that theportable device 2522 intersecting the interface location 2417. Acorresponding interface action is taken upon determination of theintersection of portable device 2522 with interface location 2417.

In one example application, the portable device is being used as aspeaker phone on a conference call and interface location 2417 enables aprocess that mutes or unmutes the portable device. While on theconference call, the observer decides to have a conversation withanother person located to the left of the observer. The observer turnsto the left to enable the conversation, while moving the portable devicethrough the interface location 2417, and in response to the intersectionof the portable device with the interface location, the speaker and/ormicrophone of the portable device is muted, thereby enabling aconversation with the other person to the left of the observer. Theobserver proprioceptive knowledge of the interface location relative tothe right shoulder of the observer allows the observer to cause theintersection without looking at the portable device or without lookingin the vicinity of the interface location. If the observer needs torefresh their knowledge of the interface location and/or its function,then the portable device can be used to augment the interface locationwith an object image that indicates the location and function of theinterface location. Furthermore, the observer's knowledge may berefreshed by other display systems such as the headsets and terminalsdescribed herein.

FIG. 33A and FIG. 33B show an example of a stereoscopic terminaloperating a proprioceptive user interface wherein an interface locationis moving with a body part of the observer and the observer isintersecting an interface location with either a left hand or a righthand. In this example, the observer retains the proprioceptive userinterface of FIG. 24, and FIG. 27-FIG. 32, and is interfacing with astereoscopic terminal similar to the block diagram of FIG. 8. As in FIG.28A, interface locations 2416 and 2417 are beyond the limited projectionspace of the terminal. In FIG. 33A, interface locations 2414 and 2415are beyond the limited projection space because the observer's head isturned to the left. In FIG. 33B, interface locations 2414 and 2415 arebeyond the limited projection space because the observer's head isturned to the right. In FIG. 33A the observer 2400 is using a finger ofthe left hand 3310 to indicate an intersection with interface location2414, and in FIG. 33B the observer 2400 is using a finger of the righthand 3320 to indicate an intersection with the same interface location2414. A process related to interface location 2414 is enabled inresponse to either the left handed intersection of FIG. 33A or the righthanded intersection of FIG. 33B.

In one example, an intersection of interface location 2414 may beconsidered a manual input that generates a manual input signal thatenables a process that toggles the rendering of object images thataugment interface locations. Thus, the observer of FIG. 33A couldintersect interface location 2414 with the right hand 3310 a first timeand augmentation of interface locations would be enabled and anyinterface location occurring within the limited projection space of theterminal 2502 would be augmented with an object image rendered by thestereoscopic terminal. A second intersection of interface location 2414with the right hand would enable a process that toggled the describedaugmentation, which in this continuing example results in disabling theaugmentation. In a similar example, the observer of FIG. 33B couldintersect interface location 2414 with a finger of the left hand andagain enable the process that toggles the augmentation of interfacelocations. Note that in the example of FIG. 33, none of the interfacelocations 2414-2417 would be augmented because they are not shown aslocated within the limited projection space of the terminal. However,applying the same example to FIG. 28A, an intersection with interfacelocation 2414 would result in enabling a process that toggles therendering of object images augmenting interface locations 2414 and 2415.In other examples, other processes can be enable based on theintersection with an interface location.

FIG. 33A and FIG. 33B show that an imaging system can determinelocations of body parts, relative locations of interface locations andphysical objects indicating intersections with interface locations eventhough the interface locations are not augmented with object images andeven though the interface locations are not located within the limitedprojection space of the stereoscopic terminal. The observer's naturalproprioceptive abilities provide for this potential advantage.Furthermore, an interface location may be intersected with any of anumber of different motions. In this example, the same interfacelocation is intersected by fingers of the left and right hands, and thesame process is enabled based on the determined intersection, ratherthan a specific motion or gesture on the part of the observer. Theobserver may also indicate the intersection with another physical objectsuch as a wand, and/or the intersection may be the result of a pointingoperation and the line of the pointing direction augmented with astereoscopic extension, as previously described. Thus, the processenabled based upon the determined intersection may be implemented by anyof a number of motions by the observer, none of which need be apredefined gesture motion.

FIG. 34 shows an example of an observer wearing a head mounted deviceobserving a reflection of the observer in a mirror with interfacelocations of the reflection augmented with object images rendered by thehead mounted device. The imaging system of the headset 2512 captures animage of the observer in the mirror, determines the location ofreference elements associated with body parts in the reflection,determines reflection interface locations relative to the body parts ofthe reflected image and augments the reflection interface locations. Inthis example, interface location 2415 is relative to the nose 2404 ofthe observer 2400, and reflection interface location 2415R is relativeto the reflection of the nose 2404R of the observer 2400R. Interfacelocation 2416 and new interface location 3414 are located relative tothe left shoulder 2406 of the observer 2400, and reflection interfacelocation 2416R and 3414R are relative to the reflection of the leftshoulder 2406R of the observer 2400R. The reflection interface locations2414R-2417R and 3414R are within the limited projection space 2514 ofthe headset and are shown and filled rectangles and a filled circlebecause they are augmented with object images rendered by projectors inthe headset 2512 to appear as if they are relative to the reflection ofthe observer. Interface locations 2416, 2417 and 3414 are beyond thelimited projection space 5814 of the headset 2512 and are shown asunfilled rectangles because they are not augmented. Interface location2415 is within the limited projection space of the headset and is shownas a filled circle and may or may not be augmented with an object imagerendered by the headset. Not augmenting interface location 2415 mayenhance the observer's perspective of the reflection because thereflection is not obscured by the rendering of an object image appearingto be between the observer and the mirror.

FIG. 35 shows an example of a perceived image of an observer wearing ahead mounted device looking at an observer reflection in a mirror,wherein the reflection is augmented with object images rendered by theheadset to indicate reflected interface locations. The observer viewsthe reflection of the observer 2400M in the mirror 3400. In thereflection 2400M, nose of the observer may be considered a reflectedbody part. The reflection 2400M is not rendered by the headset, but isan image of the observer reflected in the mirror 3400. The reflectioninterface locations 2415R, 2416R, 2471R, and 3414R are augmented withobject images rendered by the headset to appear, from the perspective ofthe observer, as reflected object images at the locations of thereflection interface locations relative to the reflection body parts ofthe observer reflection 2400M. One possible advantage of augmenting thereflection of the observer is to enable the observer to better learn orrefresh their understanding of the proprioceptive user interface. Forexample, due to the limited projection space 2514 of the headset 2512,the observer may not be able to simultaneously observer all interfacelocations of the proprioceptive user interface of FIG. 34. The distancebetween interface location 2416 and 2417 is greater than the limitedprojection space of the headset when worn by the observer. However, inthe reflection of the observer, the observer can see object imagesrendered to augment all interface locations, thereby enhancingunderstanding of the proprioceptive user interface. The observer mayalso practice intersecting interface locations by observing thereflection of the observer intersecting the reflected interfacelocations that are augmented.

Furthermore, if the interface locations are movable relative toreference elements associated with body parts of the observer, and ifthe observer may change the location of an interface locations relativeto a body part, similar to the way icons may be reorganized on a desktopof a two dimensional a graphical user interface, such as the graphicaluser interface provided by the Microsoft Windows operating system, thenthe use of the reflection of FIG. 35 has the possible advantage ofsimplifying and enhancing relocation of interface locations that mayotherwise be beyond the limited projection space of the observer,because additional interface locations may be viewed in the mirrorreflection. Changing or editing locations of interface locations mayalso include changing the body part to which the interface locations aremade relative. For example, interface location 2414 is relative to thenose of the observer, and may have been changed by the observer tointerface location 3414 which is relative to the left shoulder of theobserver. The process enabled by an intersection with interface location2414 or interface location 3414 (such as toggling augmentation ofinterface images), may remain the same even though the reference bodypart may have been changed when the observer moved the interfacelocation.

FIG. 36 shows an example of an observer observing a display terminalrendering a mirror image of the observer with interface locations of themirror image augmented with object images rendered by the displayterminal. FIG. 36 is similar to FIG. 34 in may aspects, a primarydifference is that the observer of FIG. 34 is viewing a mirror through ahead worn device and the observer of FIG. 36 is observing a displayterminal 2502 and a mirror and a head worn device are not required. Thedisplay terminal may be a 3D stereoscopic display terminal or may be atwo dimensional display rendering 2.5D image giving the observer aperception of depth.

FIG. 37 shows an example of a perceived image of an observer viewing adisplay terminal rendering a mirror image of the observer wherein themirror image of the observer is augmented with object images rendered bythe display terminal to indicate reflected interface locations. FIG. 36is similar to FIG. 34 in many aspects, a primary difference is thatimage 2400M is a mirror reflection of the observer and image 2400R is arendering of a mirror image of the observer by the display of theterminal. Also, the observer of image 2400R is not necessarily wearing aheadset. The potential advantages and benefits associated with theimages of FIG. 35 also apply to the images of FIG. 37. FIG. 36 and FIG.37 also share the benefit of nor requiring a head worn device and notrequiring a display capable of stereoscopic projection to realize thepossible benefits of FIG. 34 and FIG. 35 because the display may beeither a 3D stereoscopic display or a 2D display rendering a 2.5D image.

Further note that the benefits of a proprioceptive user interface do notrequire a display, and may be realized with a system able to determinean interface location relative to a reference element of a body part ofan observer and able to determine a physical object indicating anintersection with the interface location. Such a system may be apreviously described imaging system which may or may not be associatedwith a display system. Since the observer gains a natural understandingof the proprioceptive user interface, a display may not necessarily berequired to benefit from its application. For example, a home stereoaudio system may include an imaging system having cameras for observingthe observer and no corresponding display system. An observer within theimaging space of the imaging system and using a physical object toindicate an intersection interface location 2414 (building on a priorexample) may mute or unmute the audio system. In other words, theobserver raises his finger to intersection with a location in front ofand to the left of the observer's nose mutes or unmutes the audiosystem. This is a proprioceptive action that the user has learned andthe audio system has been provided with the programming to enable themuting and unmuting process. Similar to icons on a personal computerdesktop, the location and function of each interface location may beunique to the observer and need not be standardized across multipleobservers desiring to interface with device having a proprioceptive userinterface.

FIG. 38 shows a block diagram illustration of a system for providing aportable proprioceptive user interface able to allow an observer tointerface with a number of devices using a common proprioceptive userinterface. The observer's personal proprioceptive user interfaceconfiguration may be stored in a remote data base 3800 such as a cloudhaving information for implementing the proprioceptive user interface onany of a number of device. The database may include various body partsof the observer, reference elements, relative locations of interfacelocations and various processes to be enabled in response to detectionof an intersection with an interface location. The remote database maybe a process operating in a cloud, such as a process operating onservers coupled to the Internet. The data for implementing the personalproprioceptive user interface is communicated from the remote databaseto each device the observer interface with. The observer may have anidentification signal indicative of the observer in order that theproper database associated with the identification signal may bereceived by the device. The identification signal may be included withina Bluetooth signal generated by a cellphone carried by the observer.Other signals by other devices carried by the observer that identify theobserver are considered to be within the scope of the description. Theimaging system of the device may also capture an image of the observer,which may be used to identify the observer. An observer may interfacewith any of a number of devices including a headset device 2512, aterminal device 2502, a hand held or portable device 2522 or a homeaudio system 3810. The devices may be owned by the observer, or publicterminals or kiosks for interfacing with the observer. One possibleadvantage of the system of FIG. 38 is that the proprioceptive userinterface travels with the observer as the observer interfaces withvarious devices. The interface locations and processes enabled by theintersection with the interface locations may be constant from device todevice, even though the observer may have before never interfaced with aparticular device. As a further example, the each device may provideadditional interface locations unique to the functionality of thedevice.

Since each proprioceptive user interface may be personalized to anindividual observer, two observers may have identically locatedinterface locations but the enabled process may be different for eachobserver. For example interface location 2414 for a first observer mayenable a process that mutes or unmutes audio, while a second observermay have defined an interface location similar to interface location2414 which may enable a process for fast forwarding an audio playback.In each case, the proprioceptive user interface for each user would bereceived from the remote database 3800, and each user may make the sameproprioceptive intersection, even using the same motion gesture, andhave different process enabled (muting vs. fast forwarding in thisexample).

In another example, a first observer may have a first interface locationfor enabling a muting and unmuting in a first location relative to afirst body part, and a second observer may have a second interfacelocation for enabling the muting and unmuting in a second locationrelative to a second body part of the second observer. In this examplethe proprioceptive user interface device receives the proprioceptiveuser interface for the first observer from the remote data base andenables the muting and unmuting process in response to a first physicalobject interfacing the first interface location relative to a body partof the first observer. Similarly, the proprioceptive user interfacedevice receives the proprioceptive user interface for the secondobserver from the remote data base and enables the muting and unmutingprocess in response to a second physical object interfacing the secondinterface location relative to a body part of the second observer. Thus,in this example, two different observers may enable the muting processon a common device using two different interface locations and numerousdifferent motions or gestures to indicate the intersection with theinterface location.

FIG. 39 shows a representative flow diagram of a process for realizing aproprioceptive user interface. In step 3902 a signal is receivedidentifying the observer, which is communicated to a remote database atstep 3904. Step 3906 then receives an observer signal having informationindicative of the proprioceptive user interface associated with theobserver. Steps 3902-3906 may be skipped if the device already has theinformation indicative of the proprioceptive user interface associatedwith the observer. Step 3908 then determines the location of a body partof the observer and/or reference element associated with the body part.Step 3910 then determines an interface location relative to thedetermination of step 3908. Step 3912 determines if the interfacelocation is within the limited display space of the display system or ifthe display of a portable device is between the observer and theinterface location. If so, then step 3914 determines if augmentation ofthe interface location is enabled. If so step 3916 enables rendering ofan object image augmenting the interface location. Step 3918 thendetermines the location and optionally the orientation of a physicalobject and step 3920 determines if the physical object is indicating anintersection with the interface image. If so, then step 3922 enables aprocess based upon the determined intersection.

In one example, a method comprises: determining an interface locationrelative to a reference element associated with a body part of anobserver; enabling, from a perspective of the observer, augmentation ofthe interface location with an object image rendered by a displaysystem; disabling the augmentation of the interface location;determining a physical object indicating an intersection with theinterface location when the augmentation is disabled; and enabling aprocess based upon the determined intersection. The method furthercomprising determining the physical object indicating the intersectionwith the interface location when the augmentation is enabled, wherebythe process is enabled if the augmentation is either enabled ordisabled. The display system has a limited projection space and theaugmentation is enabled in based upon the interface location beingwithin the limited projection space and the augmentation is disabledbased upon the interface location being beyond the limited projectionspace. The display system includes an imaging system having an imagingspace at least partially beyond the limited projection space wherein thedetermining the interface location is based upon an at least one imagereceived by the imaging system and the interface location is beyond thelimited projection space. The determining of the physical objectintersecting with the interface location is based upon the at least oneimage received by the imaging system. The augmentation is disabled basedupon a manual input signal. The observer has a head and a neck and thebody part is included within at least one of the head and the neck ofthe observer. The head of the observer includes a nose and the body partis the nose of the observer. The reference element is a headset worn onand associated with the head of the observer, the headset including thedisplay system adapted to render the object image for augmenting theinterface location. The reference element includes at least one fiducialmark worn on the head of the observer and the display system includes astereoscopic display terminal adapted to render the object image foraugmenting the interface location. The display system is included withina handheld display device and the object image is rendered to appear atthe interface location from the perspective of the observer based uponthe display system being positioned between the head of the observer andthe interface location. The observer has a body and the interfacelocation is a distance from the body, the distance being such that thephysical object indicating the intersection with the interface locationdoes not result in the physical object making contact with the body ofthe observer as a result of the intersection. The observer has an eyeand an arm having an observer arm's length and the reference element isthe eye of the observer and the interface location is within theobserver arm's length of the eye of the observer. The observer has aneye and the reference element is the eye of the observer and theinterface location is between two centimeters and fifty centimeters fromthe eye of the observer. The method further comprising: determining asecond interface location relative to a second reference elementassociated with a second body part of the observer; enabling, from theperspective of the observer, augmentation of the second interfacelocation with a second object image rendered by the display system;disabling the augmentation of the second interface location; determiningthe physical object indicating a second intersection with the secondinterface location when the augmentation of the second interfacelocation is disabled; and enabling a second process based upon thesecond intersection. The observer has a head and a shoulder and the bodypart of the observer corresponds to the head of the observer and thesecond body part of the observer corresponds to the shoulder of theobserver. The reference element is a fiducial mark removably affixedwith the body part. The observer has a head and the reference element isa headset removably affixed to the head of the observer. The observerhas an eye and the reference element is the eye of the observer. Theobserver has nose and the reference element is the nose of the observer.The observer has a shoulder and the reference element is the shoulder ofthe observer. The method further comprising: determining a secondinterface location relative to the reference element; enabling, from theperspective of the observer, augmentation of the second interfacelocation with a second object image rendered by the display system;disabling the augmentation of the second interface location; determiningthe physical object indicating a second intersection with the secondinterface location; and enabling a second process based upon the secondintersection.

In another example, a method comprises augmenting a reflected image ofan observer with a reflected object image appearing at a reflectedinterface location relative to a reflected element of a reflected bodypart of the observer included within the reflected image. The reflectedimage of the observer is an image of the observer reflected by a mirror.The reflected image of the observer is rendered by a display terminal.The method further comprising augmenting an image of the observer withan object image appearing at an interface location relative to anelement of a body part of the observer.

In another example, a method in a portable device having a displaysystem comprises: determining an interface location relative to areference element associated with a body part of an observer of thedisplay system; determining the display system to be located between theobserver and the interface location; and augmenting, with the displaysystem, the interface location with an object image from a perspectiveof the observer based upon the determining of the display system to belocated between the observer and the interface location. The furthercomprising: determining the portable device indicating an intersectionwith the interface location; and enabling a process based upon theintersection. The portable device indicates the intersection with theinterface location by being placed at the interface location. Theportable device indicates the intersection based upon a manual input atthe portable device indicative of a selection of the object imagerendered on the display. The method further comprising: determining aphysical object indicating an intersection with the interface location;and enabling a process based on the intersection.

In another example, a method comprises: determining, by a deviceseparate from a head of an observer, an interface location relative to areference element associated with the head of the observer; determininga physical object indicating an intersection with the interfacelocation; and enabling a process based upon the determined intersection.The method further comprising selectively enabling and disabling, from aperspective of the observer, augmentation of the interface location withan object image rendered by a display system, whereby the process isenable if the augmentation is either enabled or disabled.

In another example, a method comprises determining an interface locationrelative to a reference element associated with a head of an observer,the interface location not augmented with an image indicative of theinterface location; determining a physical object indicating anintersection with the interface location; and enabling a process basedupon the determined intersection. The interface location is a distancefrom the head of the observer, the distance being such that the physicalobject indicating the intersection with the interface location does notresult in the physical object making contact with the head of theobserver. The method further comprising enabling, from a perspective ofthe observer, augmentation of the interface location with an objectimage rendered by a display system, whereby the process is enabled ifthe augmentation is either enabled or disabled.

In another example, a portable device comprises: a display systemconfigured to render an object image; a first imaging system receivingan at least one image of an observer observing the display system theimage having a reference element associated with a body part of theobserver; an interface location determiner determining an interfacelocation based upon the reference element of the at least one image; thedisplay system configured to render the object image at the interfacelocation from a perspective of the observer; an intersection determinerfor determining if a physical object indicates an intersection with theinterface location; and a controller configured to enable a processbased upon the determined intersection. The physical object correspondsto the portable device. The portable device further comprising a secondimaging system receiving an at least one second image including theinterface location, wherein the intersection determiner determines ifthe physical object indicates the intersection with the interfacelocation based upon the second image.

In another example, a device comprises: an observer locator configuredto locate a reference element relative to a body part of an observer; aninterface location determiner configured to determine an interfacelocation based upon the reference element; a physical object detectorfor determining a physical object indicating an intersection with theinterface location; and a controller configured to enable a processbased upon the intersection. The device further comprising: a displaysystem having a limited projection space; and an object image renderingdeterminer configured to determine that the interface location occurswithin the limited projection space, wherein the display system rendersan object image at the interface location from a perspective of theobserver based upon the interface location occurring within the limitedprojection space. The display system is included within a headset wornby the observer. The physical object detector is included within theheadset. The headset is the reference element when worn on the head ofthe observer, and the interface location determiner is included withinthe headset and the interface location determiner includes an imagingsystem for capturing an at least one image of the reference element.

In another example, a method comprises: determining a physical objectindicating a first intersection with an interface location relative to areference element associated with a body part of a first observer with afirst imaging system; enabling a process based upon the firstintersection determined with the first imaging system; determining thephysical object indicating a second intersection with the interfacelocation with a second imaging system; and enabling the process basedupon the second intersection determined with the second imaging system.The method further comprising: receiving a manual input for setting theinterface location for the relative to the reference element. The methodfurther comprising: determining a second physical object indicating asecond intersection with a second interface location relative to asecond reference element associated with a body part of a secondobserver with the first imaging system; enabling the process based uponthe second intersection determined with the first imaging system;determining the physical object indicating a second intersection withthe interface location with a second imaging system; and enabling theprocess based upon the second intersection determined with the secondimaging system.

In another example, a method comprises determining a first observer tobe within an imaging space of an imaging system; determining a firstphysical object indicating a first intersection with a first interfacelocation relative to a first reference element associated with a firstbody part of the first observer with the imaging system; enabling aprocess based upon the first intersection; determining a second observerto be within the imaging space of the imaging system; determining asecond physical object indicating a second intersection with a secondinterface location relative to a second reference element associatedwith a second body part of a second observer with the imaging system,the first interface location relative to the first observer beingdifferent from the second interface location relative to the secondobserver; and enabling the process based upon the second intersection.The method further comprising receiving from a remote database a firstobserver signal indicative of the first reference element and the firstinterface location based upon the determining the first observer to bewithin the imaging space of the imaging system; and receiving from theremote database a second observer signal indicative of the secondreference element and the second interface location based upon thedetermining the second observer to be within the imaging space of theimaging system. The imaging system includes a display system forrendering an object image and the method further comprising: augmenting,from a perspective of the first observer, the first interface locationwith the object image; and augmenting, from a perspective of the secondobserver, the second interface location with the object image.

In another example, a method comprises: determining a first observer tobe within an imaging space of an imaging system; determining a firstphysical object indicating a first intersection with an interfacelocation relative to a reference element associated with a first bodypart of the first observer with the imaging system; enabling a firstprocess based upon the first intersection; determining a second observerto be within the imaging space of the imaging system; determining asecond physical object indicating a second intersection with theinterface location relative to the reference element associated with asecond body part of the second observer with the imaging system, thesecond body part of the second observer corresponding to the first bodypart of the first observer, and the first interface location relative tothe first observer corresponding to the second interface locationrelative to the second observer; and enabling a second process differentfrom the first process based upon the second intersection.

In a summary, the disclosure also includes an abstract summary where aproprioceptive user interface defines interface locations relative tobody parts of an observer, the body parts include the head and theshoulders of the observer. The observer may enable a process associatedwith the interface location by indicating an intersection with theinterface location with a physical object, such as a wand or a finger ofthe observer. Proprioception allows the observer to know the location ofthe interface location without needing to observe the location as it isperceived as an extension of a body part of the observer. Consequently,an object image need not be rendered at the interface location by adisplay system. This provide additional display area for the displaysystem and allows for interface locations to be placed outside a limiteddisplay space of the display system. Stereoscopic display systems aredescribed, such display systems may be found in terminals and head worndevices. Also, portable devices such as cell phones may be used tooperate the proprioceptive user interface.

IV. Non-Limiting Items

The present subject matter can also be embedded in a computer programproduct, which comprises all the features enabling the implementation ofthe methods described herein, and which, when loaded in a computersystem, is able to carry out these methods. Computer program in thepresent context means any expression, in any language, code or notation,of a set of instructions intended to cause a system having aninformation processing capability to perform a particular functioneither directly or after either or both of the following a) conversionto another language, code or, notation; and b) reproduction in adifferent material form.

Each computer system may include, inter alia, one or more computers andat least a computer readable medium allowing a computer to read data,instructions, messages or message packets, and other computer readableinformation from the computer readable medium. The computer readablemedium may include computer readable storage medium embodyingnon-volatile memory, such as read-only memory (ROM), flash memory, diskdrive memory, CD-ROM, and other permanent storage and may be consideredas non-transitory computer readable medium having a stored set ofinstructions that when executed implement processes described herein.Additionally, a computer medium may include volatile storage such asRAM, buffers, cache memory, and network circuits. Furthermore, thecomputer readable medium may comprise computer readable information in atransitory state medium such as a network link and/or a networkinterface, including a wired network or a wireless network, which allowa computer to read such computer readable information.

Although specific embodiments of the subject matter have been disclosed,those having ordinary skill in the art will understand that changes canbe made to the specific embodiments without departing from the scope ofthe disclosure. The scope of the disclosure is not to be restricted,therefore, to specific embodiments or examples, and it is intended thatthe appended claims define the scope of the present disclosure.

I claim:
 1. A method comprising: locating and tracking a movement of areference element relative to a first movable body part of an observer;determining a location of an interface location positioned relative tothe reference element which is relative to the first movable body partof the observer; determining a physical object controlled by a secondmovable body part of the observer indicating an intersection with theinterface location when an object image is not rendered at the interfacelocation relative to the reference element which is relative to thefirst movable body part of the observer by an augmentation system thathad previously rendered the object image at the interface locationrelative to the reference element which is relative to the first movablebody part of the observer; enabling a process based upon theintersection which has been previously determined; determining that theinterface location occurs within a limited projection space of theaugmentation system; and augmenting the interface location relative tothe reference element which is relative to the first movable body partof the observer by rendering the object image at the interface locationwhen the interface location relative to the reference element which isrelative to the first movable body part of the observer occurs withinthe limited projection space of the augmentation system, wherein theaugmenting further includes selectively augmenting the interfacelocation relative to the reference element which is relative to thefirst movable body part of the observer based upon an input signal,wherein the process is enable if the augmentation is either enabled ordisabled based upon the input signal.
 2. The method according to claim 1wherein the augmentation system is included within a headset detachablyworn by the observer.
 3. The method according to claim 2 wherein theintersection is determined by a physical object detector included withinthe headset.
 4. The method according to claim 2 wherein the intersectionis determined by a physical object detector included within a deviceexternal to the headset.
 5. The method according to claim 4 wherein theinterface location is positioned by the device external to the headset.6. The method according to claim 5 wherein the device external to theheadset is a stationary terminal.
 7. The method according to claim 1further comprising receiving a signal indicative of the referenceelement, a position of the interface location, and the process to beenabled in response to detection of the intersection at the interfacelocation and augmenting the interface location based upon the signal. 8.The method according to claim 1 wherein the object image includes anicon indicative of the enabled process.
 9. A method comprising: locatingand tracking a movement of a reference element relative to a firstmovable body part of an observer; determining a location of an interfacelocation positioned relative to the reference element which is relativeto the first movable body part of the observer; determining a physicalobject controlled by a second movable body part of the observerindicating an intersection with the interface location when an objectimage is not rendered at the interface location relative to thereference element which is relative to the first movable body part ofthe observer by an augmentation system that had previously rendered theobject image at the interface location relative to the reference elementwhich is relative to the first movable body part of the observer;enabling a process based upon the intersection which has been previouslydetermined; determining that the interface location occurs within alimited projection space of the augmentation system; and augmenting theinterface location relative to the reference element which is relativeto the first movable body part of the observer by rendering the objectimage at the interface location when the interface location relative tothe reference element which is relative to the first movable body partof the observer occurs within the limited projection space of theaugmentation system, wherein the physical object controlled by thesecond movable body part includes a portable device having theaugmentation system.
 10. A method comprising: locating and tracking amovement of a reference element relative to a first movable body part ofan observer; determining a location of an interface location positionedrelative to the reference element which is relative to the first movablebody part of the observer; determining a physical object controlled by asecond movable body part of the observer indicating an intersection withthe interface location when an object image is not rendered at theinterface location relative to the reference element which is relativeto the first movable body part of the observer by an augmentation systemthat had previously rendered the object image at the interface locationrelative to the reference element which is relative to the first movablebody part of the observer; and enabling a process based upon theintersection which has been previously determined, wherein the referenceelement is included within an article detachably worn on the firstmovable body part of the observer.
 11. The method according to claim 10wherein the article is an article of clothing.
 12. The method accordingto claim 1 wherein the physical object controlled by the second movablebody part includes a portable device having the augmentation system. 13.The method according to claim 1 wherein the reference element isincluded within an article detachably worn on the first movable bodypart of the observer.
 14. The method according to claim 9 furthercomprising receiving a signal indicative of the reference element, aposition of the interface location, and the process to be enabled inresponse to detection of the intersection at the interface location andaugmenting the interface location based upon the signal.
 15. The methodaccording to claim 9 wherein the object image includes an iconindicative of the enabled process.
 16. The method according to claim 9wherein the portable includes a cell phone.
 17. The method according toclaim 10 wherein the augmentation system is included within a headsetdetachably worn by the observer.
 18. The method according to claim 10further comprising receiving a signal indicative of the referenceelement, a position of the interface location, and the process to beenabled in response to detection of the intersection at the interfacelocation and augmenting the interface location based upon the signal.19. The method according to claim 10 wherein the object image includesan icon indicative of the enabled process.
 20. The method according toclaim 10 wherein the physical object controlled by the second movablebody part includes a portable device having the augmentation system.